Author | Adrien Joly
Translated by | Winter rain
Planning | Ding Xiaoyun
Node.js as a back-end framework, has been widely adopted by a growing number of companies since it was first released in 2009. Its success is due to the application of the JavaScript language (also known as the Web language), a rich ecosystem of open source modules and tools, and its simple and efficient prototype API.
Unfortunately, simplicity is a double-edged sword. A simple Node .js API becomes more and more complex as it grows, and developers who lack experience with software design and best practices can quickly be overwhelmed by software entropy, occasional complexity, or technical debt.
In addition, the flexibility of the JavaScript language can easily be abused, and a normally usable prototype can quickly become an unsupportable monster while running in production. When starting a project with Node .js, it's easy to overlook best practices traditionally used with OOP languages like Java and C#, such as solid principles, which is better or worse.
I feel the pain of software entropy when I help my clients (mostly start-up companies) improve their Node .js code base, and in the open source projects I write. For example, I face increasing challenges when maintaining the Node .js application openwhyd.org that I started writing 10 years ago. I often find similar challenges in my clients' Node .js codebases: features that are being added break seemingly unrelated features, bugs become difficult to detect and fix, automated tests are challenging to write, run slowly, and fail for strange reasons...
Let's explore why some Node .js codebases are harder to test than others. And explore several techniques for writing tests that are simple, robust, and quick to check business logic. Includes dependency injection (i.e. SOLID's "D"), endorsement testing, and (spoiler warning) no mock!
1
Test the business logic
As a practical example, let's take a look at a feature in Openwhyd that hasn't been covered by automated testing: "Hot Tracks."
This feature is one of the most frequently published, liked, and played music charts by Openwhyd users in the last 7 days.
It consists of three use cases:
Display a list of tracks ;
Update the ranking when a song is posted, retweeted, liked and/or played;
Show the pop trend (i.e. up, down, or stable) of each song through changes in track ranking.
To prevent regression on the pleasant path of these three use cases, let's describe the following test cases as behavior-driven development (BDD) scenarios:
Let's imagine how you can turn your first scenario into an ideal automated test:
At this stage, this test will not pass because getHotTracks() requires a database connection, our tests are not provided, and storeTracks() has not been implemented yet.
From now on, passing the test will be our goal. To better understand why "top tracks" are difficult to test in this way, let's take a look at the current implementation.
2
Why this test fails (current)
Currently, openwhyd's popular track feature consists of several functions derived from models/tracks .js files:
getHotTracks() is called by the HotTracks API controller to get a list of sorted tracks before rendering;
updateByEid() is called when a track is updated or deleted by a user to update its popularity score;
snapshotTrackScores() is called every Sunday in order to calculate the trend for each track displayed in the following week.
Let's see what the getHotTracks() function does:
Writing unit tests for this function is complex because its business logic (for example, calculating the trend of each track) is intertwined with a data query that is sent to a global MongoDB connection (mongodb.js).
This means that in the current implementation, the only way to test Openwhyd's popular track logic is:
Test the system as a black box by sending API requests to a running Openwhyd server connected to the MongoDB server;
These functions are called directly after the dependent MongoDB database is initialized.
Both solutions need to be started and counted on the MongoDB database server. This will make our tests complex to implement and slow to run.
Conclusion: Coupling business logic to I/O (such as database queries) makes it difficult to write tests, slows down their execution, and makes them vulnerable.
3
Simulated problems
One way to avoid relying on a MongoDB database to run tests is to use what Jest calls a "mock" to simulate the database. (Or "pile," as Martin Fowler defines in Simulation is not a pile.)
Injection simulation requires Test Runner to hot-swap the dependencies used by the system under test (for example, the database client used by our server) with a fake version so that automated tests can override the behavior of that dependency.
This is totally possible.
But what if the feature in the test calls the same function multiple times to make different queries?
In this case, the simulations and tests that initialize them quickly become larger, more complex, and therefore more difficult to maintain.
More importantly, doing so means that automated testing relies on implementation details independent of business logic.
Two reasons:
mocks will be bound to the implementation of our data model, that is, whenever we decide to refactor it, we will have to override them (e.g. rename properties);
Mocks are bound to the interfaces that are replaced by dependencies, that is, we have to rewrite them whenever we upgrade mongodb to a new version, or when we decide to migrate database queries to a different ORM.
This means that even if the business logic doesn't change, sometimes we have to update our automated tests!
In our case, if we decide to simulate a mongodb dependency in a test, writing and updating the test will require more work. To avoid this, developers may be advised to upgrade dependencies, change the data model, or worse: write tests in the first place!
Of course, we'd rather save some time doing more important things, like implementing new features!
Tip: When relying on simulation to test tightly coupled code, automated tests can fail even if the business logic does not change. In the long run, mock database queries make testing more unstable and less readable.
4
Dependency injection
Based on the previous example, mocking a database query is unlikely to be a viable, long-term approach to testing business logic.
Can we abstract the dependencies between business logic and the data source mongodb as an alternative?
yes. We can decouple the feature and its underlying data acquisition logic by having the caller of the feature inject a way for the business logic to get the data it needs.
In practice, instead of importing mongodb from our model, we pass the model as a parameter so that the caller can specify any implementation of that data source at run time.
Here's how to convert the getHotTracks() function to a type expressed in TypeScript:
This way:
Different implementations of fetchRanked Traces() and fetchCorrespondingPosts() can be injected based on the execution environment of our application at getHotTracks() calls: mongodb-based implementations will be used for production, while custom memory implementations will be instantiated for each automated test;
We don't need to start the database server, nor do we need to run tests to inject simulations, we can test the logic of the model;
Automated tests do not need to be updated when the API of a database client changes.
Conclusion: Dependency injection facilitates decoupling between business logic and the data persistence layer. We can refactor tightly coupled code to make it easier to understand and maintain, and write robust and fast unit tests for it.
5
Be careful with the Ship of Ten Thousand Years
In the previous section, we learned how dependency injection can help decouple between business logic and the data persistence layer.
To prevent bugs when refactoring the current implementation, we should ensure that refactoring does not have any effect on the behavior of the feature.
To detect behavioral changes in tightly coupled code that are not adequately covered by automated tests, we can write endorsement tests. The accreditation test collects tracks in advance and performs again after the changes are implemented to check that the tracks remain unchanged. They are temporary until it is possible to write better tests (such as unit tests) for our business logic.
In our case:
On the input (or trigger) side: when an HTTP request is received by the /hot and /api/post endpoints, the "Top Tracks" feature is triggered by Openwhyd's API;
In terms of output (or track): These HTTP endpoints provide a response and may insert and/or update objects in the tracks data collection.
Therefore, we should be able to detect functional regressions by making API requests and observing changes in the resulting response and/or the state of the tracks data collection.
Note that these tests can be run as-is against Openwhyd's API because they only operate on external interfaces. Therefore, these endorsement tests can also be used as gray box tests or end-to-end API tests.
The first time we run these tests, the test runners will generate a snapshot file containing the data passed to ToMatchSnapshot() for each test assertion. Before committing these files to our version control system (e.g. git), we must check that the data is correct and sufficient for reference. Hence the name: "Accreditation Test".
Note: It is important to read the implementation of the test functions to discover the parameters and characteristics that these tests must cover. For example, the getHotTracks() function accepts a limit and skip argument for paging, and it merges the extra data obtained from the post collection. Ensure that the coverage of the endorsement test is increased accordingly to detect regressions in all key parts of the logic.
6
Problem: Same logic, different tracks
After you submit the snapshot and rerun the endorsement tests, you may find that they fail!
Jest tells us that the object identifier and date are different every time we run...
To solve this problem, we replace the dynamic value with a placeholder before passing the result to Jest's toMatchSnapshot() function:
Now that we have kept a reference to the expected output for these use cases, we can safely refactor our code and ensure that the output is consistent to run the tests again.
7
Refactor for unit tests
Now that we have an endorsement test to warn us if the behavior of the "hot tracks" feature has changed, we can safely refactor the implementation of that feature.
To reduce the cognitive load in the refactoring process we are about to begin, let's start with the following steps:
Remove all dead code and/or commented out code;
Use await on asynchronous function calls instead of passing callbacks or calling .then() on promises ;(this will greatly simplify the process of writing tests and moving blocks of code)
Add the FromDb suffix after the names of database-dependent legacy functions to make a clear distinction from the new functions we are about to introduce. (For example, rename the getHotTracks() function to getHotTracksFromDb() and the fetchRankedTracks() function to fetchRankedTracksFromDb())
It's risky to start renaming and moving code blocks based on our intuition. The risk of doing so is that the resulting code is difficult to test...
Let's go another way: write a test that clearly and unambiguously examines the behavior of the features, and then refactors the code so that the tests pass. The Test-Driven Development Process (TDD) will help us come up with a new design that makes the feature easy to test.
The test we're going to write is unit testing. As a result, they run very fast and do not require a database to start, nor do they require Openwhyd's API server. To achieve this, we will extract the business logic so that it can be tested independently of the underlying infrastructure.
Also, we don't plan to use snapshots this time. Instead, let's express exactly how human-readable features should behave, similar to early BDD applications.
Let's start with a very simple question: if there's only one track on Openwhyd, it should be listed at the top of the top hits.
So far, this test has been valid, but it has not passed because getHotTracks() is not defined. Let's provide the simplest implementation, just to get the tests to pass.
Now that the tests have passed, the third step of the TDD method suggests that we should clean up and/or refactor our code, but so far not much has been done! So let's start the second TDD iteration by writing the second test.
This test failed because getHotTracks() returns a hard-coded value in order for the first to pass the test. In order for this function to work in both test cases, let's provide the input data as an argument.
Now that our two unit tests have passed a very basic implementation, let's try to getHotTracks() closer to its actual implementation (called getHotTracksFromDb()), the implementation currently in use in production.
To maintain the purity of these tests (i.e. tests that do not produce any side effects and therefore do not run any I/O operations), the getHotTracks() function they call must be independent of the database client.
To achieve this, let's apply dependency injection: replace the poststedtracks argument (type: array of tracks) of getHotTracks() with the getTracksByDescendingScore() function, which will provide access to those tracks. This will allow getHotTracks() to call the function when data is needed. As a result, we give more control to getHotTracks() while passing on the responsibility for how to actually get the data to the caller.
Now that we've made the pure implementation of getHotTracks() closer to the real implementation, let's call it from the real implementation!
Our unit tests and accreditation tests still work, proving that we didn't break anything!
Now that the "Top Tracks" model refers to our pure "Top Tracks" feature logic as "Top Tracks," we can gradually shift the logic from the first to the second as we write unit tests.
Our next step will be to move the complete tracks data from posts with additional metadata from getHotTracksFromDb() to getHotTracks().
We observe from the production logic:
Similar to tracks, posts are fetched from the database by calling the fetchPostsByPid() function, so we will have to apply dependency injection to the function again;
The data between the track and post collections is associated by two fields, eId and pId.
Before shifting that logic, based on these observations, let's define the expected behavior of getHotTracks() as a new unit test.
In order for the test to pass, we move the call to fetchPostsByPid() and its subsequent logic, from getHotTracksFromDb() to getHotTracks().
At this point, we move all the data manipulation logic to getHotTracks(), and getHotTracksFromDb() contains only the necessary pipelines to provide it with the actual data from the database.
To get the test to pass, we only need to do one last thing: pass the fetchPostsByPid() function as an argument to getHotTracks(). For our two initial tests, fetchPostsByPid() can return an empty array.
Now that we have successfully extracted the business logic from getHotTracksFromDb() to getHotTracks() and overridden that pure logic with unit tests, we can safely remove the previously written endorsement test to prevent the function from returning: it renders the ranked tracks.
We can follow the exact same process to accomplish the remaining two use cases:
Write unit tests based on BDD scenarios,
Refactor the underlying function so that the test passes,
Delete the appropriate endorsement test.
8
conclusion
We've improved the testability and testing methods of the code base:
An example of production code was considered, which was complicated to test because the business logic was tightly coupled to database queries;
Discusses the disadvantages of relying on databases (real or simulated) when writing automated tests for logic;
Wrote endorsement tests to detect any functional regressions that might occur when refactoring logic;
Following TDD, the logic is refactored step by step using the dependency injection principle (also known as the "D" in "SOLID");
Remove the endorsement tests to support the pure, human-readable unit tests we write along the way.
Adopting these patterns and principles, such as SOLID, that are widely accepted and applied in object-oriented programming languages (OOP) can help us write better tests and make our codebase easier to maintain while maintaining the ergonomics of the JavaScript and TypeScript environments.
I would like to thank my colleague Julien Top u (SHODO's technical coach) for his review and refinement of the concepts, as well as the examples used to improve the testing methods.
About the Author
Adrien Joly, who has been a software engineer since 2006, is based in Paris, France. He works primarily for startups. He is concerned with producing software that is both useful and easy to write through regular coordination and review with technical and functional collaborators. Ten years after writing his first node .js-based full-stack web application (openwhyd.org), he still maintains it in production and uses it to practice legacy code refactoring techniques. In March 2020, Adrien Joly joined shodo, a consultancy, to grow with like-minded professionals and to practice his skills as a software craftsman. You can follow Adrien or contact him on Twitter: @adrienjoly.
https://www.infoq.com/articles/testing-legacy-nodejs-app/
Translator Profile
Winter rain, a small technical house, is now engaged in R & D process improvement and quality improvement work, focusing on R & D, testing, software engineering, agile, DevOps, cloud computing, artificial intelligence and other fields, very willing to share foreign fresh IT information and in-depth technical article translation to everyone, has translated and published "in-depth agile testing", "continuous delivery practice".