Rebirth of the Internet Overlord

{Time: March 4, 2003}

{Location: Palo Alto, California}

It is always good to be modest at any time, but being too modest can sometimes lead to misunderstandings. So facing the Kaltix trio, Ning Zimo was not modest.

The technology that the Kaltix trio said they have mastered is indeed aimed at improving Fengyu’s pagerank. However, Bing also has its own dedicated algorithm for web page weights. The essential commonality of search engines makes the integration of Kaltix technology no problem.

.

The problem is, how to bring the Kaltix trio under its banner. This is not as simple as poaching the Kaltix trio, but a "coquettish operation" on Fengyu and Bing's relative ebb and flow.

Since one thing is going down, the other is going up, Ning Zimo doesn't mind showing off his muscles.

Use lightning speed to capture three people!

Therefore, this extended scenario of big data search was no longer enough to be described as stunning in the era of 2003.

Including Hoffman, the four people in the office stared at Ning Zimo with confused expressions, which made Ning Zimo feel like he had said it before.

Indeed, what I just said is true for the four people in front of me.

Too ahead of its time!

their understanding,

Maybe I can't keep up.

Unexpectedly, Saipu suddenly opened his mouth and said cautiously, "Ning, the three of us have discussed the development of search engine technology regarding the Fengyu search engine. We thought that our vision was already very advanced, but I didn't expect that you would lay it out in front of us.

The picture is completely beyond our imagination.”

The young man named Ning in front of me is not only the famous coo of Beibao, but also the co-founder of [Lingying] and [myspace].

He has demonstrated his creativity in several projects in the past, but he did not expect that he would have unique insights beyond this era in such a professional field as search engines.

What he focuses on is not just technology, but also the key role that technology, or Internet technology, will play in the future development of human civilization.

If many people asked themselves in the past - what can search engines do, their answers would at best be better than the answers given by Ning:

"Search engines can help you find more professional and detailed answers faster."

But after today, Sepp felt that he might have a better answer for the person who asked the question.

In other words, after today, he suddenly couldn't give anyone an answer.

What can search engines do?

Ning has made it clear,

But that future,

It is indeed still far away from now.

Thinking of this, Sepp mustered up the courage to ask, "Ning, the picture you painted is definitely the most worthy development direction for search engines to explore in the future."

Nodding apologetically, Sepp said with a smile, "It's not that I'm not optimistic about the future you mentioned, but I realize that the future you mentioned is a bit far away from us. It will definitely use a lot of new technologies.

To implement, those technologies cannot be completed by a simple team.

Even Fengyu, which is currently very leading in the field of search engines, even if it has enough technology accumulation, cannot achieve the future you mentioned in the short term. How to concatenate web page files into data and use database query method

Implementing search is currently a difficult task in the short term.”

"You always have to have a dream, otherwise how can you realize it step by step?"

Ning Zimo changed his words about dreams into words. At the same time that Saipu expressed his stance, he had already thought of something deeper.

Ning Zimo understands that there are some people who cannot be completely convinced by drawing cakes alone.

I must use my whole body to deal with these young people who also have dreams!

Ning Zimo smiled confidently and took a sip of coffee to compile the key technologies of Fengyu Search into a book in his mind.

When Ning Zimo took the cup away from his mouth, the key technology of Fengyu search was sealed in his mind.

Already open!

"When it comes to search, people tend to simply think that search is just the results of crawlers crawling from the web. But in fact, search is not as simple as people think."

Ning Zimo gently placed the cup on the table, glanced at the three of them with a smile, and explained in detail:

"When we talk about big data search, its core must be a huge amount of data. This kind of volume of data storage, indexing and retrieval can no longer be realized and solved by database structures alone.

Data storage and search of large amounts of data must have a corresponding file storage and retrieval system, which must be a scalable distributed file system for large-scale data-intensive applications.

In addition to our efforts in search and crawling, the most important part of our [bing] is the core technology surrounding the bfs (bing file system) file system. The design goals of this system are similar to many traditional distributed files.

The systems have many similarities. For example, performance, scalability, reliability and availability.

However, bfs is still based on the analysis of application load conditions and technical environment. Whether now or in the future, bfs is obviously different from the early distributed file system assumptions. Therefore, we have re-examined the design of traditional file systems.

The eclectic choices derived from completely different design ideas.

First, component failures are considered normal rather than accidental. BFS needs to manage hundreds or thousands of storage machines, which are accessed by a considerable number of user terminals at the same time. The quantity and quality of BFS components result in the fact that at any given time

It may happen that some components fail to function and that some components cannot recover from their current failed state.

When we encounter various problems, such as application bugs, operating system bugs, human errors, and even problems caused by hard disk, memory, connectors, network and power failure, etc. Therefore, continuous monitoring,

Error detection, disaster redundancy and automatic recovery mechanisms must be integrated into gfs.

Second, our files are huge by usual standards. Files of several gigabytes can be very common. Each file typically contains many application objects, such as web documents.

When we need to deal with rapidly growing terabyte data sets composed of hundreds of millions of objects in the future, it is very unwise to manage small files with hundreds of millions of kb in size, although some file systems support such management.

method. Therefore, design assumptions and parameters, such as i/o operations and block size, need to be reconsidered.

Third, most files are modified by appending data to the end of the file rather than overwriting the original data. Random writing operations to the file almost do not exist in practice. Once the writing is completed, the operation on the file is

Only read, and usually sequential reading.

A large amount of data meets these characteristics, such as: extremely large data sets scanned by data analysis programs; continuous data streams generated by running applications; archived data; intermediate data generated by one machine and processed by another machine.

The processing of these intermediate data may be carried out at the same time, or may be processed later.

For this access mode for massive files, it is meaningless for the client to cache data blocks. Data append operations are the main consideration for performance optimization and atomicity guarantees.

Fourth, the co-design of applications and file system APIs improves the flexibility of the entire system. For example, we have relaxed the requirements for the BFS consistency model, which alleviates the file system's stringent requirements for applications and greatly simplifies BFS

design.

BFS also introduces atomic record append operations to ensure that multiple clients can perform append operations at the same time, without requiring additional synchronization operations to ensure data consistency."

Picking up the coffee and taking another sip, Ning Zimo was used to the stunned state of the four people around him.

Putting down the cup, Ning Zimo was no longer too lazy to explain the technical details, but he still mentioned:

"Bfs has planned four core technologies early on, which covers the first part of the distributed infrastructure. The modules include file system (file), distributed lock service (chubby) and data sequence protocol (protocol buffer).

).

The second part is the distributed large-scale data processing module, which contains the programming framework of distributed computing programs and the corresponding data query language. They may be similar to sql, but in fact it should be called dsl (domain-nguage

).

The third part is distributed database technology, which includes modules such as distributed data storage system (bigtable) and database partitioning system (sharding).

The last part, of course, includes data center optimization technologies, which include comprehensive considerations of increasing the temperature of the data center, as well as corresponding integration technologies for batteries and servers, etc.”

Ning Zimo curled his lips, frowned and said, "Search engines are a comprehensive technical consideration, but I only gave a very simple example, and its scope of application is only in web search results, and I did not mention it.

A holistic consideration of the entire ecology.

But the technical content I am talking about is far more complicated than what I am telling you now. Each single item will contain the corresponding work results of a large number of staff, and it is indeed not something that can be achieved in the short term.

But fortunately our [bing] team has clear goals.”

"You still have to have dreams, what if they come true?" Ning Zimo curled his lips and raised his eyebrows.

This lightly provocative action was seen as solemn in the eyes of the Kaltix trio.

The office was completely silent. Except for the necessary blinks, no one made any extra movement for a long time.

Probably unable to bear the heavy atmosphere, Hoffman quickly stood up and went to the door to ask someone to help him add a few cups of coffee.

Hoffman also figured it out. Ning Zimo was a bit "hot" today, so his usual quick-and-dirty style became extremely fierce. The continuous foresight and technology openly revealed will definitely make him Stanford University

The three junior students were a little overwhelmed.

When someone brought new coffee, Hoffman hurriedly greeted him, "Come, let's drink something to digest first." Ning used to be like this when he led the team at Beibao. Every time he had a new direction, he always

It is to move forward bravely in that direction.

In the past two years, I have never seen him fail. But why he will not fail, I believe you have just seen it. That is because he has achieved the ultimate in controlling every detail, and this is why we can succeed step by step.

Every victory."

Sepp was obviously the one who made the decision among the three. As soon as Hoffman finished speaking, Sepp nodded.

Although the young Ning was always speaking, Sepp always felt that his throat was a little itchy.

He coughed and looked at his companions. Seeing that they all nodded slightly, Sepp cleared his throat and said, "Ning, we have clearly seen the future you described, just as Senior Hoffman said.

, understand that you pay great attention to every detail of your plan.

I think after what you said, we only have one question left. That is, if we join [bing], what can we do for [bing] in the early stage?"

Done!

No unnecessary exclamations.

Ning Zimo was sure that this would be the result before he did it. The dimensionality reduction attack is not only about making people surrender, but it can also be used to conquer others with their abilities.

Otherwise, when the Sagittarius fleet arrives on Earth, why would there always be humans who want to become Trisolarans?

Of course, Sepp and the others didn't have such thoughts, but they were told by themselves that they couldn't find their own direction.

Now that others are already willing to join the team, Ning Zimo's attitude is naturally much gentler:

"For Bing as a whole, Mr. Lu Qi, who came from the Duhu search engine department, has enough ability to control the overall situation. But in terms of details, we need enough experts to implement the technology.

Like Fengyu, Bing also has its own pagerank algorithm. But at present, we need experts in search engine optimization. This is the main purpose why I asked Huffman to help me find candidates, and you are Huffman for us [bing]

We are looking for technical experts. So, I want to hear what expertise you have in the search field.”

Even though he knew that the core technology of Kaltix was a set of algorithms that included three technologies, even if he knew that these three technologies were quadratic extrapolation, blockrank and adaptive pagerank, Ning Zimo still had to make a sufficient gesture.

Sepp nodded and said with a smile, "Okay, Ning, let me introduce our Kaltix to you in detail."

Search engines are not what Ning Zimo is good at, but his admiration for Fengyu in his previous life drove Ning Zimo to conduct considerable research on Fengyu technology. This accumulated search knowledge allowed Ning Zimo to communicate with Saipu.

Not strenuous.

Kaltix is ​​an emerging search ranking technology developed by Sepp based on Fengyu's core page rank technology. It sorts search results based on personal interests and can increase search engine speed to five times the original speed.

It has three core technologies...

Even if he knew this, Ning Zimo felt that he couldn't calm down after listening carefully.

It’s no wonder that Fengyu in the previous life developed rapidly in 2004. It was entirely due to the acquisition and digestion of Kaltix’s core technology, and the improvement of Fengyu search by Kaltix was obvious.

Saipu's explanation made Ning Zimo feel that he had found a treasure, and he couldn't help but praise, "The core idea of ​​your technology is a search and sorting method based on user experience. This is almost the same as the "interactive experience" that I value for my products.

requirements. The core of your technology is actually 'interactive experience', which is not only the improvement of technology but also the improvement of 'interactive experience' from the user's perspective.

This kind of technology application is very convenient and practical for customers. From now on, search is no longer a machine-style feedback result, but a personalized feedback of the real needs of users. You have done an amazing thing."

"Then do you think, with our capabilities and technology, can we join [bing]?"

Seeing Sepu looking at him innocently, Ning Zimo suddenly wanted to laugh.

In any case, this sentence indicates a result.

I once again tricked Feng Yu,

The kaltix team is back!