Nani? The backend wants to return 100,000 pieces of data to me at once! And look at my 8 solutions to deal with wit

Description of the problem

• Interviewer: The backend returns 100,000 pieces of data to you at one time, how do you deal with it?
• Me: Crooked smile, what the f**k!

Problem looks

Seemingly nonsensical questions actually test the breadth and depth of a candidate's knowledge, although this situation is rarely encountered in the workplace...

• Examine how the front end processes large amounts of data
• Examine candidates' performance optimizations for large amounts of data
• Examine the candidate's way of thinking about the problem (about this, it will be said at the end of the article, everyone continue reading)
• ......

The complete code will be provided at the end of the article for everyone to better understand

Use Express to create an interface with 100,000 pieces of data

If you are not familiar with express related, you can take a look at the author's full-stack article (and complete code): "Vue+Express+Mysql full-stack project adding, deleting, modifying, checking, pagination sorting and export table function"

js复制代码route.get("/bigData", (req, res) => {
  res.header('Access-Control-Allow-Origin', '*'); // 允许跨域
  let arr = [] // 定义数组，存放十万条数据
  for (let i = 0; i < 100000; i++) { // 循环添加十万条数据
    arr.push({
      id: i + 1,
      name: '名字' + (i + 1),
      value: i + 1,
    })
  }
  res.send({ code: 0, msg: '成功', data: arr }) // 将十万条数据返回之
})
           

Click the button, send a request, get the data, and render it to the table

The HTML structure is as follows:

html复制代码<el-button :loading="loading" @click="plan">点击请求加载</el-button>

<el-table :data="arr">
  <el-table-column type="index" label="序" />
  <el-table-column prop="id" label="ID" />
  <el-table-column prop="name" label="名字" />
  <el-table-column prop="value" label="对应值" />
</el-table>

data() {
    return {
      arr: [],
      loading: false,
    };
},

async plan() {
    // 发请求，拿数据，赋值给arr
}
           

Solution 1: Render all data directly

If 100,000 pieces of data are requested to render directly, the page will freeze, which is obviously not desirable

js复制代码 async plan() {
      this.loading = true;
      const res = await axios.get("http://ashuai.work:10000/bigData");
      this.arr = res.data.data;
      this.loading = false;
}
           

Solution 2: Use timer grouping to render in batches and heaps in turn (timing loading, heap ideas)

• Normally, 100,000 data requests take between 2 seconds and 10 seconds (it may be longer, depending on the specific content of the data)
• In this way, after the front-end requests 100,000 pieces of data, it does not rush to render, and first piles and batches 100,000 pieces of data
• For example, if a pile stores 10 pieces of data, then there are 10,000 piles of 100,000 pieces of data
• Use a timer, render a bunch at a time, and render 10,000 times
• If you do this, the page will not freeze

The renderings that the user sees are as follows

Renderings

Grouping heap functions

• Let's first write a function to pile 100,000 pieces of data
• The so-called heaping is actually the idea of intercepting a certain length of data at a time
• For example, 10 pieces of data are intercepted at a time, 0~9 is intercepted for the first time, and 10~19 and other fixed-length interceptions are intercepted for the second time
• For example, the original data is: [1, 2, 3, 4, 5, 6, 7]
• Suppose we divide the pile into 3, then the result is a two-dimensional array
• i.e.: [ [1,2,3], [4,5,6], [7]]
• Then I iterate through this two-dimensional array and get the data for each item, which is each pile of data
• Then use the timer to render a little bit and a bunch of assignments

Grouping and batching heap functions (10 in a pile)

js复制代码function averageFn(arr) {
  let i = 0; // 1. 从第0个开始截取
  let result = []; // 2. 定义结果，结果是二维数组
  while (i < arr.length) { // 6. 当索引等于或者大于总长度时，即截取完毕
    // 3. 从原始数组的第一项开始遍历
    result.push(arr.slice(i, i + 10)); // 4. 在原有十万条数据上，一次截取10个用于分堆
    i = i + 10; // 5. 这10条数据截取完，再截取下十条数据，以此类推
  }
  return result; // 7. 最后把结果丢出去即可
}
           

Create timers to assign renderings in turn

For example, we assign a rendering every second

js复制代码  async plan() {
      this.loading = true;
      const res = await axios.get("http://ashuai.work:10000/bigData");
      this.loading = false;
      let twoDArr = averageFn(res.data.data);
      for (let i = 0; i < twoDArr.length; i++) {
        // 相当于在很短的时间内创建许多个定时任务去处理
        setTimeout(() => {
          this.arr = [...this.arr, ...twoDArr[i]]; // 赋值渲染
        }, 1000 * i); // 17 * i // 注意设定的时间间隔... 17 = 1000 / 60
      }
    },
           

This method is equivalent to creating many scheduled tasks to deal with in a very short time, which is too many scheduled tasks and consumes resources.

In fact, this method has the idea of pagination of large data volume

Solution 3: Use requestAnimationFrame instead of timers for rendering

Regarding the advantages of requestAnimationFrame over timers, Taoists can read this article of the author: "Easy to understand performance optimization to learn requestAnimationFrame and use case examples"

Anyway, when you encounter a timer, you can think about whether you can use the request animation frame to optimize the rendering?

If you use the request animation frame, you have to modify the code writing, the previous one does not change, the writing in the plan method can be changed, pay attention to the comment:

js复制代码async plan() {
  this.loading = true;
  const res = await axios.get("http://ashuai.work:10000/bigData");
  this.loading = false;
  // 1. 将大数据量分堆
  let twoDArr = averageFn(res.data.data);
  // 2. 定义一个函数，专门用来做赋值渲染（使用二维数组中的每一项）
  const use2DArrItem = (page) => {
    // 4. 从第一项，取到最后一项
    if (page > twoDArr.length - 1) {
      console.log("每一项都获取完了");
      return;
    }
    // 5. 使用请求动画帧的方式
    requestAnimationFrame(() => {
      // 6. 取出一项，就拼接一项（concat也行）
      this.arr = [...this.arr, ...twoDArr[page]];
      // 7. 这一项搞定，继续下一项
      page = page + 1;
      // 8. 直至完毕（递归调用，注意结束条件）
      use2DArrItem(page);
    });
  };
  // 3. 从二维数组中的第一项，第一堆开始获取并渲染（数组的第一项即索引为0）
  use2DArrItem(0); 
},
           

Solution 4 With the pagination component, the front end is paginated (each page shows a pile, and the ideas are divided into piles)

In this way, the author once encountered, the corresponding scenario at that time was that the amount of data was only dozens, and the back-end directly threw dozens of data to the front-end and let the front-end go to pagination

The reason why the backend does not do paging is: He had something to do at the time, so he went to the front end to do pagination.

• In the case of large amounts of data, this approach is also a solution
• The idea is also to intercept on the basis of all the data
• The brief code is as follows:

js复制代码getShowTableData() { 
    // 获取截取开始索引 
    let begin = (this.pageIndex - 1) * this.pageSize; 
    // 获取截取结束索引
     let end = this.pageIndex * this.pageSize; 
    // 通过索引去截取，从而展示
    this.showTableData = this.allTableData.slice(begin, end); 
}
           

For the complete case code, please see this article of the author: "The backend returns all the data at once, let the frontend intercept and display it for pagination"

In fact, this kind of big task is split into many small tasks, this way, practice, application of the idea is the way of sharding (time), in other scenarios, such as when large files are uploaded, there is also this idea, such as a 500MB large file, split into 50 small files, one is 10MB so... As for the articles uploaded by large files, then I will write them when the author is free...

Scenario 5 Table scrolling bottoming loading (scroll to the end, load a bunch more)

The point here is that we need to judge when the scroll bar bottoms. There are two main ways to judge

• scrollTop + clientHeight >= innerHeight
• or
• New MutationObserver() to observe

At present, the principle of some mainstream plug-ins on the market is roughly these two.

The author gives an example of the plugin v-el-table-infinite-scroll, which is essentially a custom directive. Corresponding npm address: www.npmjs.com/package/el-...

Of course, there are other plugins, such as vue-scroller, etc.: one meaning, no elaboration

Note that the bottoming load is also to be divided into heaps, and the 100,000 pieces of data obtained by sending a request are divided into heaps, and then every time the bottom is reached, a pile can be loaded

Use the el-table-infinite-scroll instruction step in el-table

Installation, note the version number (distinguish between vue2 and vue3)

cnpm install --save [email protected]

Register to use the directive plugin

js复制代码// 使用无限滚动插件
import elTableInfiniteScroll from 'el-table-infinite-scroll';
Vue.use(elTableInfiniteScroll);
           

Because it is a custom directive, it can be written directly on the el-table tag

js复制代码<el-table
  v-el-table-infinite-scroll="load"
  :data="tableData"
>
  <el-table-column prop="id" label="ID"></el-table-column>
  <el-table-column prop="name" label="名字"></el-table-column>
</el-table>

async load() {
    // 触底加载，展示数据...
},
           

Case code

In order to facilitate your demonstration, here the author directly attaches a case code, pay attention to the step comments in it

html复制代码<template>
  <div class="box">
    <el-table
      v-el-table-infinite-scroll="load"
      height="600"
      :data="tableData"
      border
      style="width: 80%"
      v-loading="loading"
      element-loading-text="数据量太大啦，客官稍后..."
      element-loading-spinner="el-icon-loading"
      element-loading-background="rgba(255, 255, 255, 0.5)"
      :header-cell-style="{
        height: '24px',
        lineHeight: '24px',
        color: '#606266',
        background: '#F5F5F5',
        fontWeight: 'bold',
      }"
    >
      <el-table-column type="index" label="序"></el-table-column>
      <el-table-column prop="id" label="ID"></el-table-column>
      <el-table-column prop="name" label="名字"></el-table-column>
      <el-table-column prop="value" label="对应值"></el-table-column>
    </el-table>
  </div>
</template>

<script>
// 分堆函数
function averageFn(arr) {
  let i = 0;
  let result = [];
  while (i < arr.length) {
    result.push(arr.slice(i, i + 10)); // 一次截取10个用于分堆
    i = i + 10; // 这10个截取完，再准备截取下10个
  }
  return result;
}
import axios from "axios";
export default {
  data() {
    return {
      allTableData: [], // 初始发请求获取所有的数据
      tableData: [], // 要展示的数据
      loading: false
    };
  },
  // 第一步，发请求，获取大量数据，并转成二维数组，分堆分组分块存储
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashuai.work:10000/bigData");
    this.allTableData = averageFn(res.data.data); // 使用分堆函数，存放二维数组
    // this.originalAllTableData = this.allTableData // 也可以存一份原始值，留作备用，都行的
    this.loading = false;
    // 第二步，操作完毕以后，执行触底加载方法
    this.load(); 
  },
  methods: {
    // 初始会执行一次，当然也可以配置，使其不执行
    async load() {
      console.log("自动多次执行之，首次执行会根据高度去计算要执行几次合适");
      // 第五步，触底加载相当于把二维数组的每一项取出来用，取完用完时return停止即可
      if (this.allTableData.length == 0) {
        console.log("没数据啦");
        return;
      }
      // 第三步，加载的时候，把二维数组的第一项取出来，拼接到要展示的表格数据中去
      let arr = this.allTableData[0];
      this.tableData = this.tableData.concat(arr);
      // 第四步，拼接展示以后，再把二维数组的第一项的数据删除即可
      this.allTableData.shift();
    },
  },
};
</script>
           

Renderings

Solution 6 Use unlimited loading/virtual list for presentation

What is a virtual list?

• The so-called virtual list is actually a manifestation of the front-end trick.
• It seems that all the data is rendered, but only the viewable area is actually rendered
• It's kind of like when we watch a movie, and when we watch it, it's on a movie screen, second by second.
• But in fact, the movie has two hours, if you spread out the two-hour movie, how many movie screens do you need?
• Similarly, if 100,000 pieces of data are rendered, how many DOM node elements are needed?
• So we only show it to the user, what he can see at the moment
• If the user wants to fast forward or rewind (pull-down or pull-up scrollbar)
• Then present the corresponding content on the movie screen (rendered in the viewable area)
• This achieves the effect of looking like all DOM elements and every piece of data are rendered

Regarding the front-end blinding method, if it can be used skillfully in specific work, it will greatly improve our development efficiency

Write a simple virtual list

Renderings

Here the author directly on the code, you can copy and paste it to use, the author wrote some comments to facilitate everyone's understanding. Of course, you can also go to the author's repository to take a look, the GitHub repository is at the end of the article

code

html复制代码<template>
  <!-- 虚拟列表容器，类似“窗口”，窗口的高度取决于一次展示几条数据
            比如窗口只能看到10条数据，一条40像素，10条400像素
            故，窗口的高度为400像素，注意要开定位和滚动条 -->
  <div
    class="virtualListWrap"
    ref="virtualListWrap"
    @scroll="handleScroll"
    :style="{ height: itemHeight * count + 'px' }"
  >
    <!-- 占位dom元素，其高度为所有的数据的总高度 -->
    <div
      class="placeholderDom"
      :style="{ height: allListData.length * itemHeight + 'px' }"
    ></div>
    <!-- 内容区，展示10条数据，注意其定位的top值是变化的 -->
    <div class="contentList" :style="{ top: topVal }">
      <!-- 每一条（项）数据 -->
      <div
        v-for="(item, index) in showListData"
        :key="index"
        class="itemClass"
        :style="{ height: itemHeight + 'px' }"
      >
        {{ item.name }}
      </div>
    </div>
    <!-- 加载中部分 -->
    <div class="loadingBox" v-show="loading">
      <i class="el-icon-loading"></i>
        <span>loading...</span>
    </div>
  </div>
</template>
<script>
import axios from "axios";
export default {
  data() {
    return {
      allListData: [], // 所有的数据，比如这个数组存放了十万条数据
      itemHeight: 40, // 每一条（项）的高度，比如40像素
      count: 10, // 一屏展示几条数据
      start: 0, // 开始位置的索引
      end: 10, // 结束位置的索引
      topVal: 0, // 父元素滚动条滚动，更改子元素对应top定位的值，确保联动
      loading: false,
    };
  },
  computed: {
    // 从所有的数据allListData中截取需要展示的数据showListData
    showListData: function () {
      return this.allListData.slice(this.start, this.end);
    },
  },
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashuai.work:10000/bigData");
    this.allListData = res.data.data;
    this.loading = false;
  },
  methods: {
    // 滚动这里可以加上节流，减少触发频次
    handleScroll() {
      /**
       * 获取在垂直方向上，滚动条滚动了多少像素距离Element.scrollTop
       *
       * 滚动的距离除以每一项的高度，即为滚动到了多少项，当然，要取个整数
       * 例：滚动4米，一步长0.8米，滚动到第几步，4/0.8 = 第5步（取整好计算）
       *
       * 又因为我们一次要展示10项，所以知道了起始位置项，再加上结束位置项，
       * 就能得出区间了【起始位置, 起始位置 + size项数】==【起始位置, 结束位置】
       * */
      const scrollTop = this.$refs.virtualListWrap.scrollTop;
      this.start = Math.floor(scrollTop / this.itemHeight);
      this.end = this.start + this.count;
      /**
       * 动态更改定位的top值，确保联动，动态展示相应内容
       * */
      this.topVal = this.$refs.virtualListWrap.scrollTop + "px";
    },
  },
};
</script>
<style scoped lang="less">
// 虚拟列表容器盒子
.virtualListWrap {
  box-sizing: border-box;
  width: 240px;
  border: solid 1px #000000;
  // 开启滚动条
  overflow-y: auto;
  // 开启相对定位
  position: relative;
  .contentList {
    width: 100%;
    height: auto;
    // 搭配使用绝对定位
    position: absolute;
    top: 0;
    left: 0;
    .itemClass {
      box-sizing: border-box;
      width: 100%;
      height: 40px;
      line-height: 40px;
      text-align: center;
    }
    // 奇偶行改一个颜色
    .itemClass:nth-child(even) {
      background: #c7edcc;
    }
    .itemClass:nth-child(odd) {
      background: pink;
    }
  }
  .loadingBox {
    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    width: 100%;
    height: 100%;
    background-color: rgba(255, 255, 255, 0.64);
    color: green;
    display: flex;
    justify-content: center;
    align-items: center;
  }
}
</style>
           

Use the vxetable plugin to implement a virtual list

If it is not a list, it is a table table, the author here recommends a useful UI component, vxetable, look at the name to know that it is a table-related business. This includes virtual lists.

Vue2 and Vue3 versions are supported, the performance is better, the official said: virtual scrolling (up to 5W columns, 30W rows)

Powerful!

Official website address: vxetable.cn/v3/#/table/...

Renderings

The effect is silky

Install the usage code

Note the installation version, the version used by the author is as follows:

cnpm i xe-utils [email protected] --save

main.js

js复制代码// 使用VXETable
import VXETable from 'vxe-table'
import 'vxe-table/lib/style.css'
Vue.use(VXETable)
           

The code aspect is also very simple, as follows:

html复制代码<template>
  <div class="box">
    <vxe-table
      border
      show-overflow
      ref="xTable1"
      height="300"
      :row-config="{ isHover: true }"
      :loading="loading"
    >
      <vxe-column type="seq"></vxe-column>
      <vxe-column field="id" title="ID"></vxe-column>
      <vxe-column field="name" title="名字"></vxe-column>
      <vxe-column field="value" title="对应值"></vxe-column>
    </vxe-table>
  </div>
</template>

<script>
import axios from "axios";
export default {
  data() {
    return {
      loading: false,
    };
  },
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashuai.work:10000/bigData");
    this.loading = false;
    this.render(res.data.data);
  },
  methods: {
    render(data) {
      this.$nextTick(() => {
        const $table = this.$refs.xTable1;
        $table.loadData(data);
      });
    },
  },
};
</script>
           

Solution 7 Enable multi-threaded Web Worker for operation

In this case, using Web Worker to open another thread to manipulate the code logic is not particularly beneficial (if using a virtual scrolling list plugin)

But it can also be regarded as an extended idea, when interviewing, you can talk about it, mention it.

Taoists who are not familiar with Web Workers, you can take a look at the author's previous article: "Performance Optimization: Using the vue-worker plug-in (based on Web Worker) to open multi-threaded computing to improve efficiency"

Option 8 Take precautions and prevent problems before they occur

The following is the author's humble opinion, for reference only...

• After all the above solutions are said, it is not over.
• In fact, this topic examines not only the breadth and depth of the candidate's knowledge, but also the candidate's way of thinking about how to deal with the problem, which is particularly important!
• The author has been a candidate to apply for a job, and I have also been an interviewer to interview. As far as programmer development work is concerned, technical knowledge points are not familiar with and can be quickly learned, such as documents, Google, Baidu, technical exchange groups, relevant colleagues can provide certain support
• What is more important is to look at the candidate's way of thinking, thinking mode
• Imagine, the two candidates have similar levels of strength, but one only knows to bury his head in hard work, do it when he has work, and do not think about it; The other is when working hard, he will also look up at the starry sky, analyze how to work to complete the task cost-effectively, and pay attention to the process and result
• In this case, which is more popular?

If the author is a candidate, after the author has said the above 7 options, I will add the eighth plan: take precautions and prevent problems before they occur

Scenario simulation

The interviewer casually looked at my resume in his hand, and shouted strangely: "Boy, the back-end wants to return 100,000 pieces of data to you at once, how do you deal with it?" ”

I raised my eyebrows and smiled crookedly: "After the above 7 solutions are stated, I think we can fundamentally solve similar problems." That is, the eighth plan, we must take precautions and prevent problems before they occur. ”

"Oh?" The interviewer was puzzled and slowly put down my resume: "I would like to hear about it." ”

I replied unhurriedly: "In the specific development work, when we receive a requirement, during the technical review, we have to discuss the more appropriate technical solution with the backend." The problem is that the backend wants to return me 100,000 pieces of data at once, the point is not on 100,000 pieces of data, but on why the backend does this? ”

The interviewer looked up, reflected my figure in his pupils, and listened carefully.

I said word by word: "In addition to ** the business really needs this kind of solution ** (if it is requested by the customer, then there is nothing to say, just do it), the back-end does this for roughly two reasons, the first is that he does not understand the SQL limit statement, but this is basically impossible, and the second is that he has something to do and writes it casually." Therefore, it is necessary to communicate with him, from the perspective of the long request time of the large data volume interface, and the poor performance caused by too many DOM element rendering, and the maintainability of the project, I believe that as long as the correct communication, this unreasonable situation can be avoided from the root. ”

The interviewer suddenly asked slyly: "What if after communication, the back-end does not give you pagination?" How do you do it? Hi Hi! Your communication is not effective! How do you deal with it! People don't listen to you! As if he thought the question was tricky, he folded his arms over his chest and leaned back in his chair, letting out a weird laugh of defiance, waiting to see the unanswerable, embarrassed smile that was about to bloom on my face.

I snorted coldly in my heart: Carving insect tricks...

I stared into the interviewer's eyes and said seriously: "If communication at work is not effective, either it is a problem with my own communication language, which I will pay attention to, and constantly improve my communication skills and way of speaking, or..."

My voice raised three points: "There is a problem with this person I communicate with!" He works to fish and be lazy and slippery! Set in one's ways! Embarrassment of others! High! Opinionated! In this case, I will find my direct supervisor to intervene, because this is no longer a problem of project requirements, but a problem of basic literacy of employees! ”

After a pause for a second, my voice softened a little: "However, but I believe that there are absolutely no such people among our company's employees, and each of them is an excellent employee with strong ability and good attitude." After all, our company has a long reputation in the industry, and I am also admired for this. You're right? ”

The interviewer's eyes flashed with shock, he didn't expect that I actually kicked the ball to him again, but in order to maintain his image, he quickly regained his composure, but his facial muscles couldn't stop trembling slightly.

"Of course, the company is full of talent." The interviewer answered casually.

I added: "In fact, in the work, the front-end as a role closer to the user needs to communicate with colleagues in various positions, such as back-end, product, UI, testing, etc. We need to improve work efficiency, complete projects, realize their own value, and create benefits for the company through reasonable communication, which I think every employee needs to do and must do. ”

The interviewer shouted again: "The kid's performance is okay, you are hired!" A month's salary of 2200, bring your own computer, no company and no money, 007 work system, can't steal company snacks, and..."

Me: Ada...

Locate performance bottlenecks

Intuitively, the large amount of data is the main performance bottleneck of page rendering, but as a developer, you still have to rely on objective facts and cannot do things based on feelings. So how to locate the performance bottleneck of page lag?

In fact, there are some tools on the front end that can evaluate the performance of the website, such as lighthouse, chrome's devtool performance, and the following is mainly used to locate performance problems with these two tools.

The following is a page rendering of 4000 pieces of data, trying to use lighthouse and chorme's performance to locate the performance overhead.

Lighthouse gives optimization suggestions

First of all, through the website performance indicators given by the Lighthouse tool, the definition of several key indicators can refer to Web Vitals. As shown in the following figure:

Through this tool, we can get an approximate score of the current page performance, which is based on the scores of each indicator according to a certain weight ratio series conversion, is a comprehensive evaluation result, the lower the score, the worse the performance. The calculation of the 19 points of the current page performance is as follows:

The score of Lighthouse's each run fluctuates, which has a certain relationship with the current network speed; But it doesn't matter, the most important thing is to see the optimization suggestions it gives.

For the current page, Lighthouse's optimization suggestions are shown in the following figure:

Because it is a lighthouse running in a local development environment, some optimization items such as compressed js do not need to be paid attention to, we mainly look at the red box marked a few items, it can be said that these items are the main reasons for poor performance, and we need to focus on optimization.

• Reduce the work of the main thread
• The main thread consumes 9.4s, and the main overhead includes:
• Execution of 14 long tasks
• The execution of the pure JS part takes 7.2s
• Avoid excessive DOM numbers
• The page has 24530 DOM elements, including a DOM rendered by tables, a DOM rendered by a map Marker, etc.; Excessive DOM will cause page operation to freeze, you can refer to Why too many DOM elements on the web page cause page freezing
• Avoid large network loads
• Because the local development environment runs the result, the code is not compressed can be ignored, the real concern is the request response time of the interface, because the amount of data returned by the interface is about 48KB, resulting in the interface taking about 1.3s, and the backend interface needs to be optimized

Through Lighthouse's optimization suggestions, summarize the main problems of the current page:

• The main thread takes a long time and includes 14 long tasks
• The number of DOM pages is too large
• The interface response takes a long time, up to 1.4s

For the problem of the long time of the main thread, Lighthouse will give the time taken by each part of the entire browser rendering process, but it will not tell us in detail where each part takes time, which is the strength of Chrome devtool's Performance panel.

Performance panel positioning takes time

Performance is used to record and analyze all the activities of our application while it is running, and it presents multi-dimensional data that can help us locate performance issues well. Among them, using the Performance panel main item, the content related to the main thread of the browser is displayed, including:

• View the entire process of browser rendering, including data loading, HTML parsing, style parsing calculation, JS loading execution, composite layers, drawing, and other stages
• By viewing the call stack information of the JS script execution process and the corresponding time consumption, it is easy to locate the places where the performance time is long

The figure below is the result of using the chrome Performance panel.

It can be seen that it takes nearly 10s to render the page, of which JS execution takes about 7.6s, and it can be seen from the call stack of JS execution that it is mainly the _next that costs 3.4s and the fulfilled parts of 3.94s under Microtasks:

• The _next part is mainly the rendering time of the page component, including the rendering of each row of the table, and the execution time of the map custom marker component
• The fulfilled part is to add the marker and popup components generated by the _next to the map, because it is using dom rendering, after adding the map, it involves the html parsing and rendering process of marker and popup.

As can be seen from the figure, the fulfilled part of the time consumption is mainly to add markers and their popups to the map in a loop and complete the rendering.

Combined with the analysis of lighthouse and Performance panels, the main reasons that affect page performance in the front-end can be located are:

• Table rendering takes too long due to the large amount of data, because it renders full at one time
• Map Marker rendering is rendered in DOM form, which involves the parsing and rendering of the DOM, which has poor performance at large data volumes
• The table and map marker are rendered at the same time, causing the _next part to take a long time and blocking the execution of other important processes in the future

Performance issues are torn down

For the performance problems of the above located page, the optimization solution thought of:

• Tables do not render all the data at once
• Marker rendered in DOM form is replaced by canvas rendering
• The table and map are rendered separately, and the map marker is rendered in pieces

Big data list rendering

Shard rendering

For big data table rendering, the first thing that comes to mind is fragmented rendering, which simply means that the list of large data quantities is divided into n groups for rendering, and one group is called a data slice. Its design ideas:

Establish a queue to add rendered tile data to the render queue with a timer.

Note that the shards that have already been rendered in the render queue only consume node diff time and are not re-rendered. The demo looks like this:

code.juejin.cn/pen/7142775…

There are several major issues with fragmented rendering:

• The total render time increases because of the fragmentation via timer, there are gaps
• The data is ultimately the result of full rendering, resulting in an excessive number of DOMs
• Fast pull-up to load the splash screen, which can be solved by using requestAnimationFrame, referring to high-performance rendering of 100,000 pieces of data (time sharding)

Virtual lists

Virtual lists are another common solution to large number of table renderings, and their design ideas are:

Only the content in the viewable area is rendered, and the content in the non-viewable area is not rendered or partially rendered (commonly known as buffer)

This scheme to deal with filtering out the visual area or adding buffer data from a large amount of data and rendering, mainly according to scrolling events to filter, most of the other data content will not render the real DOM, which greatly reduces the rendering time of the table and the number of page DOMs, and the performance improvement is very considerable.

A picture is worth a thousand words, and the picture comes from here.

There are many introduction schemes for virtual lists in the community, and the specific implementation details will not be introduced too much here, you can refer to the following two articles:

• Spend three hours fully mastering shard rendering and virtual lists~
• Hands-on teaching you to write React virtual lists

The project is optimized according to the virtual list scheme with reference to the demo provided by antd.

Taking the 4,000 pieces of data mentioned above as an experiment, the effect of using a virtual list optimization without optimizing the map data is as follows:

It can be seen that the execution time of the _next part is less than 1s, and the execution time of JS is reduced to 4621ms, which significantly improves performance.

Map large data volume rendering

Map marker elements are rendered in DOM mode, and there is no problem when the amount of data is small, but as the amount of data increases, the performance of DOM rendering becomes more and more difficult, resulting in page stuttering or even crashing. Since the map uses leaflet, the solution that comes to mind is to use canvas to draw, as for why canvas is improved compared to dom rendering performance, you can refer to the "Su Bingtian" in the HTML industry of this article - detailed explanation of Canvas's superior performance and practical application].

Because the marker interaction in the map is relatively simple, click the marker to display the corresponding popup, so the project uses the plugins Leaflet.Canvas-Markers officially recommended by leaflet to generate marker, the plugin needs to use pictures to set marker, please refer to the demo for details.

Map marker rendering optimization:

jsx复制代码<MapContainer {...props}>
   {
       stops.map((stop) => {
         return <CustomMarker key={stop.id} data={stop} />;
        })
    }
 </MapContainer>
 
 // CustomMarker实现：
 function CustomMarker(props) {
     ...
     // react-leaflet提供的Marker是以Dom形式渲染的
     <Marker
      ...
      position={position}
    >
      <Popup
        {...props}
      >
        ... // popup内容
      </Popup>
    </Marker>
 }
           

After optimization:

jsx复制代码 // 主页面的render部分有关地图部分
 <MapContainer {...props}>
   <CanvasMarkers data={stops} />
 </MapContainer>
 
 // CanvasMarkers实现
 import 'leaflet-canvas-marker';
 function CanvasMarkers({data}) {
  
      const map = useMap();
      const canvasLayerRef = useRef();
      const icon = window.L.icon({
      iconUrl: '图片地址',
      iconSize: [8, 8],
      iconAnchor: [4, 4],
    });
      useEffect(() => {
        if (!data.length) {
          canvasLayerRef.current?.onRemove(map);
          return;
        }
        canvasLayerRef.current = window.L.canvasIconLayer({}).addTo(map);

        const canvasMarkers = [];
        for (let i = 0, len = data.length; i < len; i++) {
          const { lat, lng } = data[i];
          const marker = window.L.marker([lat, lng], {
            icon,
          }).bindPopup(popupHtml);
          canvasMarkers.push(marker);
        }
        canvasLayerRef.current?.addLayers(canvasMarkers);
      }, [markers, icon, map]);

      return null;
 }
           

As can be seen from the rendered DOM structure, the plugin eventually renders all Marker elements in the same canvas.

In order to compare the effects in the previous section, the same experiment with 4000 pieces of data, while using virtual list optimization, using canvas to render map elements after optimization is shown in the following figure:

It can be seen that the execution time of the fulfilled part is reduced to less than 100ms, and the execution time of the entire JS is reduced to 1102ms, which shows the performance improvement of canvas rendering.

Long task segmentation

According to the analysis of the performance panel without any optimization of the previous 4,000 data, the long tasks executed by js accounted for about 7.4s, accounting for nearly 75% of the entire main process, mainly caused by the simultaneous rendering of tables and maps, which seriously affected the execution of subsequent tasks. One benefit of long task segmentation is that it reduces the execution time of a task, freeing up execution time for later tasks.

Drawing on the shard rendering scheme of the list, can the table rendering and map data be rendered in sequential order, and the map data adopts the shard rendering mechanism?

This is technically possible, and it is acceptable for a page to let users see more important data first, and then gradually render the content of the entire page.

Therefore, the following two aspects of technical optimization have been made:

• 1. Table data and map data are rendered successively, and the table is displayed before the map elements
• JSX copies the code
• Table rendering before map data rendering, key code setTableData(tableData); setTableData provides methods for react's useState setTimeout(() => { // 100ms delay initialization of map data setMarkers(stops); setMarkers provides methods for react's useState }, 100);
• 2. Fragmented rendering of map elements
• The CanvasMarkers component is transformed as follows:
• JSX copies the code

import 'leaflet-canvas-marker'; function CanvasMarkers({data}) { ... const [data, setData] = useState([]); // 地图marker数据，分割渲染核心逻辑 const sliceData = useCallback((list, index: number = 0, num = 100) => { const endIdx = Math.ceil(list.length / num); if (index === endIdx) { return; } setTimeout(() => { // 每200ms执行一个批次的渲染 const toBeRenderList = list.slice(index * num, (index + 1) * num); setData(toBeRenderList); console.log(toBeRenderList, toBeRenderList.length); sliceData(list, index + 1, num); }, 200); }, []); useEffect(() => { const len = markers.length; if (len === 0) { canvasLayerRef.current?.onRemove(map); return; } canvasLayerRef.current = window.L.canvasIconLayer({}).addTo(map); // 分割大小的一个简单设置策略 const sliceLen = len > 100000 ? 2000 : len > 50000 ? 1000 : 500; sliceData(data, 0, sliceLen); }, [markers, map, sliceData]); // 只关注当前data的内容，为其生成L.marker useEffect(() => { if (!data.length) { return; } const canvasMarkers = []; for (let i = 0, len = data.length; i < len; i++) { const { lat, lng } = data[i]; const marker = window.L.marker([lat, lng], { icon, }).bindPopup(popupHtml); canvasMarkers.push(marker); } canvasLayerRef.current?.addLayers(canvasMarkers); }, [data, map]); return null; }           

We use the same 50,000 pieces of data to experiment, and under the premise of the above two optimizations, the performance panel results before the optimization without long task segmentation are shown in the figure below

After the optimization methods of the above two methods, the effect obtained is as follows:

You can see that the total rendering time of the page has become longer, why?

As mentioned earlier, fragmented rendering increases the total rendering time, because each batch of data is rendered asynchronously at the specified timer interval, so the total rendering time is lengthened.

But this is not our focus, we pay more attention to the TBT (total blocking time) of the page: reduced from 2690ms to 1045ms, improving efficiency by more than 60%; In addition, the page data is rendered in the table and the first batch of map data is rendered in about 4.5s, which is about 1.6s less than the 6.1s of full rendering, and the performance improvement is obvious.

Optimize performance

The 4,000 pieces of data mentioned at the beginning of the article are compared, and the results after optimization in the above three ways are as follows:

You can see a significant performance improvement:

• The JS execution time was reduced from 7453ms to 1466ms
• The total page time has been reduced from 9979ms to 2999ms, including the total time spent on sharding, and it is theoretically possible to compare the time after the first batch of map data is rendered

The optimized page opening speed almost reaches the effect of opening in seconds, and the user experience is greatly improved.

As an aside, performance optimization is a topic that is a common topic in the front-end and needs to be focused on by front-end development, especially in the field of visualization. I hope this article sharing will be helpful to everyone, and I hope that experienced bigwigs will share this knowledge.

summary

Effective communication stems from a problem-solving mindset and, in most cases, is more important than the technical knowledge at the moment

• Website effect demonstration address: ashuai.work:8888/#/bigData
• GitHub repository address: github.com/shuirongshu...