Simply put, the best soccer formation is the one that utilizes your players' strengths and minimizes the impact of your opponents' strengths. The best formation for a team with strong defensive backs and a weak striker (Arsenal, perhaps, which uses a 4-2-3-1) will probably look very different from a team with strong strikers and weak defensive backs (say, Manchester City, which uses a 4-1-4-1). Similarly, it might be best to employ one formation against a team with strong midfield play that controls the ball well, and a completely different one against a team that doesn't aim to control the ball and plays mostly on the counter-attack.
That said, formation theory has been evolving for more than a century. The first formations were often extremely attacking-heavy--the 2-3-5 (with two defenders, three midfielders, and five forwards) dominated for a half-century. Yet as players became more mobile and executed more passing moves, accepted wisdom has coalesced around formations that have 3-5 defenders, 3-5 midfielders, and 1-3 forwards (and a goalkeeper, of course).
In the 1990s and early 2000s, the soccer world thought it had the answer: the 4-4-2. Among the top leagues, the 4-4-2 was so ubiquitous that many considered it the one true formation, no further questions required. The key to the 4-4-2 was in its balance. The four defenders were enough to repel attacks from the opposing team, the four midfielders brought width to the attack while also allowing play through the middle, and the two forwards allowed for interplay in the attack and gave defenders fits. Midfielders were of utmost importance in this formation. The two central midfielders must cover basically the entire length of the field (box-to-box) for defense and attack. They receive passes from the team's defenders and start the attack by either passing out wide to the wide midfielders or up the pitch to the forwards. Meanwhile, the wide midfielders must be very fast to take the ball up the sides of the pitch and send in crosses to the forwards. Many clubs still use the 4-4-2, including most notably Leicester City in their storybook championship season of 2015-16.
Yet, in the late 2000s the 4-2-3-1 formation was able to usurp the 4-4-2 and claim the title of best formation. The key to the 4-2-3-1 is in its flexibility. Instead of the four midfielders in a line as with the 4-4-2, instead the 4-2-3-1 uses two defensive midfielders whose job is to control the midfield, as well as three attacking midfielders who link the defensive midfielders and the lone striker. When matched against a 4-4-2, a team playing a 4-2-3-1 can often use its extra midfielder to control the midfield and thus, the game. Importantly, the midfielders in a 4-2-3-1 play closer to the center of the pitch than the wide midfielders in the 4-4-2, so a 4-2-3-1 relies on wingbacks to provide width in the attack. It's no coincidence that the 4-2-3-1 surged in popularity when wingbacks started to become faster and more versatile. Now that many top clubs have wingbacks who can beat defenders on the dribble, send in accurate crosses, and rush back to defend, a 4-2-3-1 formation can provide a varied attack without sacrificing much defensive solidity. Additionally, the defense in a 4-2-3-1 is bolstered by the defensive midfielders, who are often staggered vertically to play slightly different roles. The more defensive of the two, often called the "destroyer," sits just in front of the central defenders and shields them from attack. Meanwhile, the more offensive of the two, known as the "double pivot," serves as the focal point for the offense, as he or she receives the ball from the defenders and passes to the attacking midfielders or the wingbacks.
Surely, the 4-2-3-1 isn't the best soccer formation for everyone. As previously mentioned, Leicester City were able to use a 4-4-2 with great success in 2015-16, basing their strategy on surrendering the midfield and scoring from counter attacks. Chelsea, who lead the Premier League as of writing, use a 3-4-3, which adds an extra central defender for more defensive solidity. And Pep Guardiola, considered the best coach in the world, uses a 4-1-4-1 with Manchester City that has overwhelmed defenses with attacking midfielders. Yet if a team has fast wingbacks and defensively-sound midfielders, a 4-2-3-1 is a hard formation to beat.