The new Microsoft flight The simulator is a technological marvel that sets a new standard for the genre. But to recreate a world that feels real and alive and contains billions of buildings all in the right places, Microsoft and Asobo Studios have relied on the work of multiple partners.
One of these is the small Austrian startup Blackshark.ai from Graz which, with a team of only about 50 people, has recreated every city and country in the world with the help of artificial intelligence and enormous computing resources in the cloud.
Before the launch of the new Flight Simulator, we sat down with Blackshark co-founder and CEO Michael Putz to talk about the collaboration with Microsoft and the broader vision of the company.
Blackshark is actually a spin-off of the game studio Bongfish, the creator of World of Tanks: Frontline, Motocross Madness and the Stoked snowboard game series. As Putz told me, it was actually Stoked who started the company on the road to what would become Blackshark.
"One of the first games we did in 2007 was a snowboard game called Stoked and S Stoked Bigger Edition, which was one of the first games with a full 360 degree mountain where you can use a helicopter to fly and drop, land anywhere. and get off, "he explained. "The mountain itself was procedurally built and described - and the placement of vegetation obstacles, other snowboarders and small animals was also procedurally done. Then we moved on to racing, filming, driving genre, but we still had this idea of positioning and positional descriptions at the back of our minds. "
Bongfish returned to this idea when he worked on World of Tanks, simply because of how time-consuming it is to build such a huge map where each rock is placed by hand.
Based on this experience, Bongfish began building an internal AI team. That team used a number of machine learning techniques to build a system that could learn from how designers build maps and then, at some point, build their own AI-created maps. The team actually ended up using it for some of their projects before Microsoft hit the scene.
"By pure chance, I met someone from Microsoft who was looking for a studio to help them with the new Flight Simulator. The main idea of the new Flight Simulator simulator was to use Bing Maps as a playing field, as a map, as a background, "Putz explained.
But Bing Maps' photogrammetry data has only produced exact 1: 1 replicas of 400 cities - for the vast majority of the planet, though, that data doesn't exist. Microsoft and Asobo Studios needed a way to build the rest.
This is where Blackshark comes in. For Flight Simulator, the studio reconstructed 1.5 billion buildings from 2D satellite imagery.
Now, while Putz says he met the Microsoft team by accident, there is something more to that. In the past, there was a Bing Maps team in Graz, which developed the first cameras and 3D versions of Bing Maps. And while Google Maps won the market, Bing Maps actually beat Google with its 3D maps. Microsoft then launched a research center in Graz, and when it closed, Amazon and others came in to capture local talent.
"So it was easy for us to fill positions like a PhD in roofing reconstruction," Putz said. "I didn't even know it existed, but it was exactly what we needed and we found two.
"It's easy to see why reconstructing a 3D building from a 2D map would be difficult. Understanding the exact profile of a building is also not easy.
"What we basically do in Flight Simulator is we look at the areas, the 2D areas and then we discover the footprints of the buildings, which is actually a computer vision task," Putz said. "But if a building is obstructed by a tree shadow, we actually need machine learning because then it is no longer clear what is part of the building and what is not due to the shadow overlap - but then machine learning completes. the rest of the building. This is a very simple example. "
While Blackshark has been able to rely on some other data as well, including photos, sensor data, and existing map data, it needs to determine the height of the building and some of its features based on very little information.
The next obvious problem is figuring out the height of a building. If there is existing GIS data, the problem is easy to fix, but for most areas of the world that data simply does not exist or is not immediately available. For those areas, the team takes the 2D image and looks for hints in the image, such as shadows. To determine the height of a building based on a shadow, however, the time of day is required, and the Bing Maps images are not actually time-stamped. For other use cases the company is working on, Blackshark has it and that makes things a lot easier. And that's where machine learning comes in again.
"Machine learning takes a slightly different path," Putz noted. "Look at the shadow too, we think - because it's a black box, we don't really know what it's doing. But also, if you look at a flat roof, like a skyscraper against a shopping mall. Both have mostly flat roofs, but the furniture on the roof is different on a skyscraper than on a shopping mall. This helps AI learn when you label it right. "
Also, if the system knows that the average height of a mall in a given area is usually three stories, it can work with that.
One thing Blackshark is very open about is that his system will make mistakes, and if you buy Flight Simulator, you will see that there are glaring mistakes in the way some buildings are positioned. In fact, Putz told me he believes one of the project's biggest challenges was convincing the company's development partners and Microsoft to allow them to use this approach.
"You are talking about 1.5 billion buildings. To these numbers, you can no longer ask traditional questions and answers. And traditional finger pointing like a Halo level or something where you say "this pixel is no good, fix it" doesn't really work if you develop on a statistical basis like you do with AI. So it could be that 20% of the buildings are shut down - and that's actually the case I guess in the flight simulator - but there is no other way to tackle this challenge because outsourcing to hand model 1.5 billion buildings is , only from a logistical point of view level and also budget level, not feasible ".
Over time, that system will also improve and as Microsoft streams a lot of data to the game from Azure, users will certainly see changes over time.
Labeling, however, is still something the team has to do simply to train the model, and it's actually an area where Blackshark has made a lot of progress, although Putz wouldn't talk too much about it because it's part of the secret sauce of the company and one of the main reasons why it can do all this with around 50 people.
"Data labels were not a priority for our partners," he said. "And so we used our live labeling to pretty much label the entire planet from two or three guys [...] It puts a very powerful tool and user interface in the hands of data analysts. Basically, if the data analyst wants to detect a ship, he tells the learning algorithm what the ship is and then gets the immediate output of the detected ships in a sample image. "
From there, the analyst can then train the algorithm to further improve the detection of a specific object such as a ship, in this example, or a shopping mall in Flight Simulator. Other geospatial analytics firms tend to focus on specific niches, Putz also noted, while the company's tools are independent of the type of content analyzed.
And this is where Blackshark's broader vision comes into play. Because while the company is now gaining acclaim for its work with Microsoft, Blackshark is also working with other companies to reconstruct city scenes for self-driving simulations, for example.
"Our broader vision is a near real-time digital twin of our planet, specifically the planet's surface, which opens up a trillion use cases where traditional photogrammetry like Google Earth or what Apple Maps is doing isn't. it helps because they are only simplified for photos based on simple geometric structures. For this we have our loop where we extract intelligence from aerial data, which could be 2D images, but could also be 3Dpoint counts, which are already doing another project. And then we're viewing the semantics. "
The semantic ones, which describe the building in very precise detail, have a big advantage over photogrammetry: the shadow and light information are essentially embedded in the images, making it difficult to rekindle a scene realistically. Since Blackshark knows everything about that building he is building, he can also place windows and lights in those buildings, creating surprisingly realistic night scenes in Flight Simulator.
Point clouds, which aren't used in Flight Simulator, are another area Blackshark is focusing on right now. Point clouds are very difficult for humans to read, especially once you get very close to them. Blackshark uses its artificial intelligence systems to analyze point clouds to find out how many stories a building has.
"The whole company was founded on the idea that we have to have a huge advantage in technology to get there, and especially from video games, where huge productions like Assassin's Creed or GTA are reaching the capacity limits by having thousands of people working there, who it is very difficult to scale, very difficult to manage on continents and in a product delivered on time. It was clear to us that more automated or semi-automated steps were required to do so. "
And although Blackshark has found its start in the field of gaming - and while it's working on that with Microsoft and Asobo Studios - it's actually not focused on games, but on things like autonomous driving and geographic analysis. Putz noted that another good example of this is Unreal Engine, which started out as a game engine and is now everywhere.
"For me, having been in the gaming industry for a long time, it's so encouraging to see, because when you develop games, you know how revolutionary the technology is compared to other industries," Putz said. "And when you look at simulators, from military simulators or industrial simulators, they always look like shit compared to what we have in driving games. And the time has come when gaming technologies are spreading off the game stack and helping all those other industries. I think Blackshark is one of those examples to make this possible. "
