Time Travel: 2014

Harley Price couldn't help but nod after hearing Eclair Kilcaja's words.

Because what Eclair Kilcaja said does make sense.

Compared with their predecessors who selflessly dedicated themselves to overcoming obstacles, scientific researchers of the same generation are not so enthusiastic.

Sometimes different scientific research teams in the same country often dig holes in each other out of economic interests, honor and other factors.

Not to mention LIN HUI, a scientific researcher in a foreign country.

Why should we wishfully believe that the clues explained by LIN HUI must be correct?

Harley Price feels that even if the clues left by LIN HUI are correct, they may not all be shortcuts.

Some are even complete detours.

In this case, it is more reliable to think about things yourself.

Just listen to Eclair Kilkaja continue: "According to the rules obtained from our previous research.

There is no relationship between the previous input and the next input of the neural network.

There is no way to process sequence data where the input before and after is related information.

The technical route mentioned in the generative summary algorithm of LIN HUI clearly stated that the text information should be serialized and marked through vectors before further processing.

In this case, I feel that the algorithm proposed by LIN HUI is definitely not a general neural network.

The most likely application of LIN HUI in the generative summary algorithm is the recurrent neural network.

After all, the structure of recurrent neural networks is very suitable for processing sequence information. "

Eclair Kilcaja's words made Harley Price's eyes light up, but soon new doubts arose.

Harley Price asked: “A recurrent neural network needs to be fed more than just the current sequence of data.

You also need to enter the information about the hidden layer parameters of the recurrent neural network at the previous moment.

In this way, the correlation information between sequences can be well processed.

But it gives me the feeling that the neural network used in the LIN HUI algorithm has the shadow of a recurrent neural network.

But it seems to be a little different from the traditional recurrent neural network? "

Eclair Kilcarga muttered: "That's true. General recurrent neural networks are suitable for processing sequence structures, but they are not good at processing long sequence structures..."

While he was pondering, Eclair Kilcaja suddenly thought of something and shouted:

"I know, it must be the LSTM neural network!"

Harley Price was startled by Eclair Kilcarga's sudden shout.

However, the LSTM neural network mentioned by Eclair Kilcarga also made his eyes shine.

Harley Price knew what Eclair Kilcarga called LSTM neural networks.

The more accurate name of LSTM neural network should be called "long short-term memory neural network".

This is a special type of recurrent neural network.

Compared with ordinary recurrent neural networks, long short-term memory neural networks are not sensitive to gap length in applications.

This is an advantage of long short-term memory neural networks, allowing them to perform well in processing longer sequences.

Harley Press carefully recalled the algorithm characteristics described in the technical roadmap of LIN HUI and the actual performance of the Nanfeng APP software.

The technical route mentioned in the generative summary algorithm of LIN HUI clearly stated that the text information should be serialized and marked through vectors before further processing.

The long short-term memory neural network can handle long sequences of text.

The algorithm tinkered with by LIN HUI has extremely high accuracy when processing text summarization.

A major feature of the long short-term memory neural network when dealing with practical problems is its high accuracy.

Nanfeng APP, developed by LIN HUI based on the generative summary algorithm, can only process one news summary at a time.

The disadvantage of long short-term memory neural network is that its structure is relatively complex and there are disadvantages in parallel processing.

If the algorithm of LIN HUI is just one aspect, it is consistent with the characteristics of long short-term memory neural network.

Harley Price might have thought it was just a coincidence.

But now, three small-probability events have come together.

Harley Price doesn't think it's just a coincidence.

He felt more and more that Eclair Kilcaja’s inference was correct, and couldn’t help but sigh:

"I asked why the neural network used in the LIN HUI algorithm has the shadow of a recurrent neural network but is somewhat different from the traditional recurrent neural network. It turns out that it uses a long short-term memory neural network.

Eclair Kilcarga really has you! I can actually think of this and this direction.

To be honest, I thought the neural network characteristics used in the LIN HUI algorithm were a bit weird at first.

But I really haven’t thought about the direction of long-short-term memory neural network..."

Eclair Kilcarga can understand why Harley Price didn't think of long-short-term memory neural networks in the first place.

In recent years, long short-term memory neural networks have been mainly used for speech recognition.

At this stage, ordinary researchers would never think of using long short-term memory neural networks for text summarization.

However, theoretically speaking, it is completely feasible to use long short-term memory neural network for text recognition.

But for the time being, Eclair Kilcarga is not sure how to apply long-short-term memory neural networks to text recognition.

This still requires some time of research to explore.

Harley Price did not expect that there should be another reason for the long short-term memory neural network.

Because long short-term memory neural networks are nothing new.

Hochreiter and Schmidhuber proposed the long short-term memory neural network in 1997.

It has been nearly twenty years since today, which can be said to be a long time ago.

Although long short-term memory neural networks may have certain advantages when processing long sequences of text.

But in fact, the long short-term memory neural network was not originally proposed for text processing.

The reason why the long short-term memory neural network was proposed at that time was to deal with the gradient disappearance and gradient explosion problems that may be encountered when training traditional recurrent neural networks.

In machine learning, when training artificial neural networks using gradient-based learning methods and backpropagation.

Sometimes you encounter vanishing gradient and exploding gradient problems.

Neither scenario is what researchers want to see happen.

After gradient disappearance or gradient explosion occurs, the original deep learning cannot be deepened at all, and can only be said to be shallow learning.

In some extreme cases, not to mention shallow learning, even the most basic machine learning cannot be achieved.

In summary, gradient vanishing and gradient exploding problems can greatly reduce the training efficiency of deep learning using neural networks.

The problem of vanishing gradient and exploding gradient is also an extremely difficult problem.

Researchers related to neural networks noticed the vanishing and exploding gradient phenomena in 1991.

This problem has been alleviated to some extent after the emergence of long short-term memory neural networks.

However, the problems of vanishing gradient and exploding gradient have not been completely solved.

In addition to using long short-term memory neural networks, there are several other ways to deal with the problems of vanishing and exploding gradients (such as multi-level hierarchies, using faster hardware, using other activation functions, etc.). limitations of each.

In short, the problems of gradient disappearance and gradient explosion have not been completely solved.

Today, the problems of gradient disappearance and gradient explosion have become a dark cloud in the sky of machine learning.

This problem has seriously restricted the development of machine learning.

Thinking of this, Eclair Kilcaja couldn't help but feel a little emotional.

I don’t know when this problem will be completely solved by whom? ? ?

Eclair Kilcaja suddenly felt that there was no need for him to be so serious about the LIN HUI algorithm?

Problems such as gradient disappearance and gradient explosion have not been completely solved for more than 20 years.

Isn’t there no one in a hurry? At least no one seems to be in a hurry?

Why do I have to compete with an algorithm like LIN HUI?

Eclair Kilcaja suddenly felt tired.

However, faced with excited colleagues, Eclair Kilcarga was not ready to back down.

Eclair Kilcarga: “I’m not sure yet that the long short-term memory neural network is used in the LIN HUI algorithm.

It can only be said that the characteristics of the neural network used in the LIN HUI algorithm are somewhat similar to the long short-term memory neural network.

Whether it is true or not remains to be verified.

Speaking of which, it was a real loss that those senior executives had a bad fight with the MIT Natural Language Processing Text Summarization Research Group.

As far as I know, Eve Carley and others used recurrent neural networks when studying extractive text summarization algorithms.

It’s just that the specific type of recurrent neural network they used is not yet clear.

But anyway, I think it would be a great help for us if we could have help from MIT. "

Harley Price: “It’s a problem, but it’s not a big problem.

What country M is most indispensable for is research institutions.

Some time ago, I heard from Nick that Professor Jules of Princeton University was working on a recurrent neural network project.

Maybe we could work with Princeton University? "

Eclair Kilcaja: “Well, are you sure you want to deal with those arrogant math guys at Princeton?

Do they look at us the same way we look at liberal arts students?

And if we cooperate with them, who will take the lead? How are the research results divided? "

Harley Price: “It doesn’t matter what they think of us.

A group of mathematicians are now working on recurrent neural networks. It is not certain who has the advantage?

As for who will take the lead, let’s decide later. Those who have mastered everything will be respected. "

Eclair Quircaga: "Then go ahead and get in touch. Anyway, I'm too lazy to negotiate with that old bald ass Jules."

Harley Price: "Well, actually I don't want to contact Jules either..."

Eclair Kilcaja: "Then you still come up with this bad idea?"

Harley Price said wickedly: "Maybe we can call Asile Velasquez to sell the patent to that LIN HUI..."

Eclair Kilcarga: "That's a great idea!"