Ideas for Linux Performance Tuning

The Linux operating system is an open source product, as well as an open source software practice and application platform. Under this platform, there are countless open source software supports, such as our common Apache, Tomcat, MySQL, php, etc. The biggest concept of open source software is freedom, Open, then Linux as an open source platform, the ultimate goal is to achieve the best performance of the application at the lowest cost through the support of these open source software. Therefore, when it comes to performance issues, what is mainly achieved is the best combination of linux operating system and applications.

An overview of performance issues

The performance of the system refers to the effectiveness, stability and response speed of the operating system to complete tasks. Linux system administrators may often encounter problems such as system instability and slow response speed. For example, if a web service is built on Linux, the web page cannot be opened, and the opening speed is slow. When encountering these problems, some people will Complaining that the Linux system is not good, in fact, these are superficial phenomena. When the operating system completes a task, it is closely related to the system’s own settings, network topology, routing devices, routing policies, access devices, and physical lines. Problems in any link will affect the performance of the entire system. Therefore, when there is a problem with a Linux application, a comprehensive investigation should be carried out from the application program, operating system, server hardware, network environment, etc., to locate the part where the problem occurs, and then focus on solving it.

In terms of applications, operating systems, server hardware, network environment, etc., the two aspects that affect the performance the most are the application programs and the operating system, because the problems in these two aspects are not easy to detect and are highly concealed. As long as there is a problem in the hardware and network, it can usually be located immediately. The following mainly explains the performance tuning ideas of the operating system, and the application needs to be treated in detail.

The following introduces the general ideas and methods for optimizing Linux from four aspects: factors affecting Linux performance, personnel involved in performance analysis, system performance optimization tools, and system performance evaluation standards.

1. Factors Affecting Linux Performance

2.1 System Hardware Resources

1．CPU

The CPU is the foundation of the stable operation of the operating system. The speed and performance of the CPU largely determine the overall performance of the system. Therefore, the more CPUs and the higher the main frequency, the better the server performance. But that’s not quite the case.

At present, most CPUs can only run one thread at a time, and a hyper-threaded processor can run multiple threads at the same time. Therefore, the hyper-threading feature of the processor can be used to improve system performance. Under Linux, hyperthreading is only supported by running SMP kernels, however, the more CPUs installed, the less performance gain you get from hyperthreading. In addition, the Linux kernel will recognize multi-core processors as multiple separate CPUs. For example, two 4-core CPUs will be treated as 8 single-core CPUs under the Linux system. However, from a performance point of view, two 4-core CPUs and 8 single-core CPUs are not completely equivalent. According to the test conclusions drawn by authoritative departments, the overall performance of the former is 25% to 30% lower than that of the latter.

Applications that may have CPU bottlenecks include DB servers, dynamic Web servers, etc. For such applications, the configuration and performance of the CPU should be placed in the main position.

2. Memory

The size of the memory is also an important factor affecting the performance of Linux. If the memory is too small, the system process will be blocked, and the application will become slow and even lose response; if the memory is too large, resources will be wasted. The Linux system uses physical memory and virtual memory. Although virtual memory can alleviate the shortage of physical memory, it occupies too much virtual memory, and the performance of the application will be significantly reduced. To ensure the high-performance operation of the application, the physical memory It must be large enough; but too large physical memory will cause a waste of memory resources. For example, on a Linux operating system with a 32-bit processor, more than 8GB of physical memory will be wasted. Therefore, to use larger memory, it is recommended to install a 64-bit operating system and enable Linux’s large memory kernel support.

Due to the limitation of the addressing range of the processor, on a 32-bit Linux operating system, a single process of an application can only use a maximum of 4GB of memory. Since then, even if the system has a larger memory, the application cannot “share” it. The solution is to use a 64-bit processor and install a 64-bit operating system. Under a 64-bit operating system, the memory usage requirements of all applications can be met with almost no restrictions.

Applications that may have memory performance bottlenecks include NOSQL servers, database servers, cache servers, etc. For such applications, the memory size should be placed in the main position.

3. Disk I/O performance

The I/O performance of the disk directly affects the performance of the application. In an application with frequent reads and writes, if the disk I/O performance is not satisfied, the application will stagnate. Fortunately, today’s disks have adopted many methods to improve I/O performance, such as the common disk RAID technology.

A disk group formed by RAID technology is equivalent to a large hard disk. Users can perform operations such as partitioning and formatting, creating a file system, etc. on it. It is exactly the same as a single physical hard disk. The only difference is the I/O performance ratio of the RAID disk group. A single hard disk is much higher, and the security of data is also greatly improved.

Depending on the disk combination, RAID can be divided into RAID0, RAID1, RAID2, RAID3, RAID4, RAID5, RAID6, RAID7, RAID0+1, RAID10 and other levels. Commonly used RAID levels are RAID0, RAID1, RAID5, RAID0+1, Here is a brief introduction.

RAID 0: Improves disk performance and throughput by gluing multiple hard drives into a larger-capacity hard drive group. This method is low-cost, requires at least two disks, but has no fault tolerance and data recovery functions, so it can only be used in environments that do not require high data security.

RAID 1: that is, disk mirroring. By mirroring the data of one disk to another disk, the reliability and repairability of disk data are guaranteed to the greatest extent, and it has high data redundancy, but the disk utilization rate is only 50%. %, therefore, the cost is the highest, and it is mostly used in occasions where important data is saved.

RAID5: Disk segmentation and parity check technology is used, which improves system reliability. RAID5 has high read efficiency and average write efficiency, requiring at least 3 disks. A disk failure is tolerated without affecting data availability.

RAID0+1: Combining RAID0 and RAID1 technologies becomes RAID0+1, which requires at least 4 hard disks. In addition to the data distributed on multiple disks in this way, each disk has its own mirror disk, which provides full redundancy, allows a disk failure without affecting data availability, and has fast read/write capabilities.

By understanding the performance of each RAID level, you can choose a suitable RAID level according to the different characteristics of the application, so as to ensure that the application achieves the best performance in terms of disks.

4. Internet broadband

Various applications under Linux are generally network-based, so network bandwidth is also an important factor affecting performance. A low-speed and unstable network will block access to network applications, while a stable and high-speed network bandwidth can Ensure that the application runs uninterrupted on the network. Fortunately, today’s networks are typically gigabit bandwidth or fiber optic networks, and bandwidth issues are gradually reducing the impact of bandwidth issues on application performance.

2.2 Operating system related resources

The performance optimization based on the operating system is also multi-faceted, which can be measured from several aspects such as system installation, system kernel parameters, network parameters, and file systems. The following is a brief introduction.

1. System installation optimization

System optimization can start from the installation of the operating system. When installing the Linux system, the partition of the disk and the allocation of SWAP memory directly affect the running performance of the system in the future. For applications with low security requirements, the disks can be made into RAID 0; for applications with high data security and no special requirements for reading and writing, the disks can be made into RAID 1; for higher requirements on read operations, For applications that have no special requirements for write operations and ensure data security, you can choose RAID 5; for applications that require high read and write requirements and high data security requirements, you can choose RAID ¹⁰⁄₀₁. In this way, different RAID levels are set according to different application requirements, and the system is optimized at the bottom layer of the disk.

With the decrease of memory price and the increasing memory capacity, the setting of virtual memory SWAP no longer requires the so-called virtual memory to be twice the physical memory. However, the setting of SWAP cannot be ignored. According to experience, if If the memory is small (the physical memory is less than 4GB), the size of the SWAP swap partition is generally set to be twice the size of the memory; if the physical memory is greater than 8GB and less than 16GB, you can set the SWAP size to be equal to or slightly smaller than the physical memory; if the memory size is more than 16GB, In principle, SWAP can be set to 0, but this is not recommended, because setting a certain size of SWAP still has a certain effect.

2. Kernel parameter optimization

After the system is installed, the optimization work is not over. Next, the system kernel parameters can be optimized. However, the optimization of the kernel parameters should be considered in combination with the applications deployed in the system. For example, if the system deploys an Oracle database application, it is necessary to share the system memory segment (kernel.shmmax, kernel.shmmni, kernel.shmall), system semaphore (kernel.sem), file handle (fs.file-max) ) and other parameters to optimize settings; if a web application is deployed, network parameters need to be optimized according to the characteristics of the web application, such as modifying network kernel parameters such as net.ipv4.ip_local_port_range, net.ipv4.tcp_tw_reuse, net.core.somaxconn, etc. .

3. file system optimization

The optimization of the file system is also a focus of system resource optimization. The optional file systems under Linux include ext2, ext3, ReiserFS, ext4, and xfs. According to different applications, choose different file systems.

The Linux standard file system starts from VFS, then ext, and then ext2. It should be said that ext2 is the standard file system on Linux, and ext3 is formed by adding logs on the basis of ext2. From VFS to ext4, the design idea is not There are too many changes, all of which are based on the design concept of super block and inode in the early UNIX family.

The XFS file system is an advanced journaling file system. XFS provides low-latency and high-bandwidth access to file system data by distributing and processing disk requests, locating data, and maintaining cache consistency. Therefore, XFS is extremely scalable and robust. , has the advantages of excellent logging function, strong scalability, and fast write performance.

At present, ext4 and xfs are the mainstream file systems on the server side. How to choose a suitable file system needs to be comprehensively determined according to the characteristics of the file system and the needs of the business.

2.3 Application software resources

Application optimization is actually the core of the entire optimization project. If there is a bug in an application, even if all other aspects reach the optimal state, the performance of the entire application system is still low. Therefore, the optimization of the application is a performance optimization process. The top priority, which puts forward higher requirements for program architects and program developers.

1. Personnel involved in analyzing system performance =====================================================

3.1, Linux operation and maintenance personnel

In the process of performance optimization, Linux operation and maintenance personnel undertake very important tasks. First, Linux operation and maintenance personnel must understand and master the current operating status of the operating system, such as system load, memory status, process status, CPU load and other information. This information is the basis and basis for detecting and judging system performance; secondly, Linux operation and maintenance personnel also grasp the hardware information of the system, such as disk I/O, CPU model, memory size, network card bandwidth and other parameter information, and then synthesize the information based on this information. Evaluate the usage of system resources; thirdly, as a Linux operation and maintenance personnel, you must also master the usage of system resources by applications. A more in-depth point is to understand the operating efficiency of applications, such as whether there are program bugs, memory For problems such as overflow, by monitoring system resources, you can find out whether there is an abnormality in the application. If there is a problem with the application, you need to immediately report the problem to the program developer, and then improve or upgrade the program.

Performance optimization itself is a complex and tedious process. Linux operation and maintenance personnel can only optimize server performance in a targeted manner only by understanding system hardware information, network information, operating system configuration information and application information, which requires Linux operation and maintenance. The maintenance personnel have sufficient theoretical knowledge, rich practical experience and a mind to analyze problems carefully.

3.2 System Architecture Designer

The second type of person involved in system performance optimization is the application architect. If the Linux operation and maintenance personnel find that the performance of the application is affected by the execution efficiency of the application after comprehensive judgment, then the program architecture designer should intervene in time to gain an in-depth understanding of the program running status. First, the system architecture designer should track and understand the execution efficiency of the program. If there is a problem with the execution efficiency, find out where the problem occurred; secondly, if there is a problem with the architecture design, then immediately optimize or improve the system architecture. Design better application system architecture.

3.3 Software developers

The last link of system performance optimization involves program developers. After Linux operation and maintenance personnel or architecture designers find program or structural bottlenecks, program developers should immediately intervene to make corresponding program modifications. When modifying the program, the execution efficiency of the program should be used as the benchmark, the logic of the program should be improved, and the code should be optimized in a targeted manner. For example, Linux operation and maintenance personnel find an SQL statement in the system that consumes a lot of system resources, grab this SQL statement to execute, and find that the execution efficiency of this SQL statement is too poor, which is caused by the low execution efficiency of the code written by the developer. , which requires this information to be fed back to the developer. After the developer receives this question, they can optimize the SQL in a targeted manner, thereby optimizing the program code.

It can be seen from the above process that the general process of system performance optimization is: first, Linux operation and maintenance personnel check the overall status of the system, mainly from the five aspects of system hardware, network equipment, operating system configuration, application architecture and program code. Comprehensive judgment, if it is found to be a system hardware, network device or operating system configuration problem, the Linux operation and maintenance personnel can solve it according to the situation; if it is found to be a program structure problem, it needs to be submitted to the program architecture designer; if it is found to be a program code execution problem , and leave it to the developer for code optimization. This completes a system performance optimization process.

1. Tuning summary =================

System performance optimization is a wide-ranging, tedious, and long-term work. Finding the root cause of performance problems is often the most difficult part. Once the cause of the problem is found, the performance problem will be solved. Therefore, the idea of ​​problem solving becomes very important.

For example, in a website system under the Linux system, users report that the website access speed is very slow and sometimes cannot be accessed.

In response to this problem, the first step is to detect the network. You can use the ping command to check whether the domain name resolution of the website is normal, and at the same time, whether the delay of pinging the server address is too large, etc. In this way, the network may be excluded first. If there is no problem with the network, then go to the second step to check the memory usage of the Linux system. Because the website response speed is slow, it is generally related to the memory. Use commands such as free and vmstat to determine whether the memory resources are in short supply. If there is no problem with the memory resources, go to the third step to check the load status of the system CPU. You can comprehensively judge whether the CPU is overloaded by the output of commands such as sar, vmstat, and top. If there is no problem with the CPU, continue Enter the fourth step, check whether there is a bottleneck in the disk I/O of the system. You can check the read and write performance of the disk through commands such as iostat and vmstat. The thing is to check if there is a problem with the program itself. Through this kind of thinking, layer-by-layer detection, step-by-step investigation, performance problems “have nowhere to hide”, and it becomes very simple to find the link where performance problems occur.