Time Travel: 2014

Follow Lin Hui’s explanation in the supplementary content in the paper.

Under the traditional training mechanism, the idea of ​​generating a text summary model is:

Corpus training → model

After introducing the pre-training mechanism according to Lin Hui’s idea.

The idea of ​​​​generating the text summary model is:

Corpus pre-training → pre-training model → fine-tuning → model

There is nothing wrong with this idea in itself.

But when Eve Carly faced this new model to generate ideas, her mind was full of problems.

What kind of pre-training methods should be introduced in specific applications to achieve twice the result with half the effort?

What kind of pre-training model is the target of pre-training?

How should we understand the "fine-tuning" of the pre-trained model?

The first two questions arise from Lin Hui's theories.

The third problem is some questions arising from the language explanation.

Although Eve Kali has been working hard to learn Chinese from Mina Kali recently.

But Chinese is obviously not something that can be learned quickly in a short period of time.

How should we understand the "micro" of Lin Hui's so-called "fine-tuning" of the pre-trained model?

Just a little adjustment?

Or is it that the so-called "micro" is just because Lin Hui himself despises the difficulty of this matter.

Eve Carly thinks it should be the latter.

It's unlikely to be a minor adjustment.

Why does Eve Carly think this way?

Eve Carley feels that models involving text summarization are often extremely complex.

A formal model involves an extremely large number of parameters.

What's more, what about the pre-trained model produced by pre-training?

This rough model, which is generated prior to the formal model, may have more complex parameters.

Of course, this is just a little speculation from Eve Carley.

Only Lin Hui himself may have the real answers to these questions.

Since coming to Lin Hui's side.

Originally, Eve Carly thought her problems would gradually diminish.

But the reality is that the problems are becoming more and more numerous.

At least Eve Carly never had any doubts about the questions just now when she was in the United States.

But Eve Kali is not discouraged.

In scientific research, raising questions is always more important than solving them.

Eve Carly knew it very well, although she had more doubts at this time than she had in the United States.

But it doesn't matter, at least the questions she raises now are closer to the nature of technology than those in the past.

And this is academic growth.

Not all is lost for Eve Carley.

Originally, she had always been curious about how Lin Hui, who was almost unknown in terms of text summarization before, was able to overtake in a corner in a short period of time.

After all, the construction of language models often takes a lot of time.

But now that I know about this pre-processing done by Lin Hui.

Eve Carly feels that this problem does not seem to be a big problem.

Follow the idea of ​​the pre-training mechanism proposed by Lin Hui in the supplementary content of the paper.

Although training is still required after the introduction of the pre-training mechanism.

It even seems like the steps are a little more complicated.

But Eve Carley estimates that training with a corpus of the same size

Training that introduces a pre-training mechanism can save at least 50% of the time compared to conventional training.

Introducing pre-training processing methods into model training will improve efficiency.

The reason here is easy to understand by analogy with learning examples.

Under normal circumstances, it is obviously more efficient to master the commonality of knowledge and then overcome difficulties than to study step by step.

In the same way, during machine learning, it will also improve efficiency by letting the machine grasp the commonalities of the data and then process the remaining labeled data.

Lin Hui was once an absolute genius in the eyes of Eve Carly.

According to Eve Carly, the focus of genius is not "talent" but "talent"

Everyone seems to know that they need to find the door when leaving the room, but they can't find the way.

The genius is the one who strolls to the door and gently opens the door under the blank gaze of everyone.

When everyone is facing the bottleneck of the extractive summary algorithm and cannot find a way out of the text summary room.

LINHUI appeared at just the right time, and in everyone's confusion, it opened a brand new door called "generative text summarization".

Looking at it now, Eve Carly feels that her previous understanding is still almost meaningless.

The fact is that Lin Hui is not only a genius in an absolute sense, but also a well-deserved strongman.

If what Lin Hui described in the supplementary content of the paper is true.

What is such a person if not a strong person?

It is no exaggeration to say that the introduction of pre-training is a revolution in the traditional corpus training method.

This will greatly assist the training of language models.

Eve Carly has a hunch that after the introduction of pre-training, the field of traditional natural language processing is expected to fully enter the era of neural network learning.

If this can really be done.

That would be a contribution of unprecedented significance.

You must know that what Lin Hui has done is not just pre-training.

Eve Carly noticed that Lin Hui’s description of pre-training in the paper was pre-training based on the idea of ​​transfer learning.

What is transfer learning?

Transfer learning can use existing knowledge to learn new knowledge.

The core of this idea is to find similarities between existing knowledge and new knowledge to draw inferences.

In the field of machine learning, it is too expensive to learn the target directly from scratch.

With the help of transfer learning, you don’t have to be so troublesome.

Many times we can use existing relevant knowledge to assist in learning new knowledge as quickly as possible.

For example, if you already know the C language, you can learn C++ by analogy;

Once you have learned Greek, you can learn English by analogy.

Everything in the world has something in common. After reasonably looking for the similarities between them.

Using this bridge to help learn new knowledge can save you a lot of new troubles.

If it is indeed based on this idea.

After pre-trained data commonality learning.

When performing additional learning on non-commonly labeled data.

If pre-training has the ability to learn by drawing inferences due to the introduction of transfer ideas.

Then it may take less time to learn from non-commonly labeled data.

What does less time mean?

It means an unprecedented improvement in efficiency.

Eve Carly had never quite understood how Lin Hui suddenly rose to prominence.

Now she has made up her mind.

However, in this case, Eve Carly felt that some of the previous estimates of Lin Hui were a bit conservative.

Training on a corpus of the same size can save at least 70% of the time by introducing a pre-training mechanism based on transfer ideas than conventional training.

This data is quite astonishing.