# Finally, Differentiable Physics is Here!

Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. A few episodes ago, we discussed a new research
work that performs something that they call differentiable rendering. The problem formulation is the following:
we specify a target image that is either rendered by a computer program, or even better, a photo. The input is a pitiful approximation of it,
and now, because it progressively changes the input materials, textures, and even the
geometry of this input in a 3D modeler system, it is able to match this photo. At the end of the video, I noted that I am
really looking forward for more differentiable rendering and differentiable everything papers. So, fortunately, here we go, this new paper
introduces differentiable programming for physical simulations. So what does that mean exactly? Let’s look at a few examples and find out
together! Imagine that we have this billiard game, where
we would like to hit the white ball with just the right amount of force and from the right
direction, such that the blue ball ends up close to the black spot. Let’s try it. Well, this example shows that this doesn’t
happen by chance, and we have to engage in a fair amount of trial and error to make this
happen. What this differentiable programming system
does for us is that we can specify an end state, which is the blue ball on the black
dot, and it is able to compute the required forces and angles to make this happen. Very close. But the key point here is that this system
is general, and therefore can be applied to many-many more problems. We’ll have a look at a few that are much
more challenging than this example. For instance, it can also teach this gooey
object to actuate itself in a way so that it would start to walk properly within only
2 minutes. The 3D version of this simulation learned
so robustly, so that it can even withstand a few extra particles in the way. The next example is going to be obscenely
powerful. I’ll try to explain what this is to make
sure we can properly appreciate it. Many years ago, I was trying to solve a problem
called fluid control, where we would try to coerce a smoke plume or a piece of fluid to
take a given shape. Like a bunny, or a logo with letters. You can see some footage of this project here. The key difficulty of this problem is that
this is not what typically happens in reality, of course, a glass of spilled water is very
unlikely to suddenly take the shape of a human face, so we have to introduce changes to the
simulation itself, but at the same time, it still has to look as if it could happen in
nature. If you wish to know more about my work here,
the full thesis and the source code is available in the video description, and one of my kind
students has even implemented it in Blender. So, this problem is obscenely difficult. So you can now guess what’s next for this
differentiable technique…it starts out with a piece of simulated ink with a checkerboard
pattern, and it exerts just the appropriate forces so that it forms exactly the Yin-Yang
symbol shortly after. I am shocked by how such a general system
can perform something of this complexity. Having worked on this problem for a while,
I can tell you that this is immensely difficult. Amazing. And hold on to your papers, because it can
do even more. In this example, it adds carefully crafted
ripples to the water, to make sure that it ends up in a state that distorts the image
of the squirrel in a way that a powerful and well-known neural network sees it not as a
squirrel, but as a goldfish. This thing is basically a victory lap in the
paper. It is so powerful, it’s not even funny. You can just make up some problems that sound
completely impossible and it rips right through them. The full source code of this work is also
available. By the way, the first author of this paper
is Yuanming Hu, his work was showcased several times in this series, in one of the earlier
videos, we showcased his amazing Jello simulation that was implemented in so few lines of code,
it almost fits on a business card. I said it in a previous episode, and I will
say it again. I can’t wait to see more and more papers
in differentiable rendering and simulations. And as this work leaves plenty of room for
creativity for novel problem definitions, I’d love to hear what you think about it. What else could this be used for? Solving video games faster than other learning-based
techniques? Anything else? Let me know in the comments below. What a time to be alive! This episode has been supported by Weights
& Biases. Here you see a beautiful final report on one
of their projects on classifying parts of street images, and see how these learning
algorithms evolve over time. Weights & Biases provides tools to track your
experiments in your deep learning projects. It can save you a ton of time and money in
these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. Make sure to visit them through wandb.com/papers
or just click the link in the video description and you can get a free demo today. Our thanks to Weights & Biases for helping
us make better videos for you. This episode has been supported by Weights
& Biases. Weights & Biases provides tools to track your
experiments in your deep learning projects. It can save you a ton of time and money in
these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. It is really easy to set up, so much so that
they have made an instrumentation for this exact paper we have talked about in this episode. Have a look here! Make sure to visit them through wandb.com/papers
or just click the link in the video description and you can get a free demo today. Our thanks to Weights & Biases for helping
us make better videos for you. Thanks for watching and for your generous
support, and I’ll see you next time!

### 100 thoughts on “Finally, Differentiable Physics is Here!”

• January 28, 2020 at 7:33 pm

Came here for the impressive papers, stayed for the positive and inspiring commentary.

• January 28, 2020 at 9:17 pm

Excellent as always!!!

• January 28, 2020 at 9:17 pm

Anyone ever tried to apply an AI onto a coding problem? Like I want an AI, that writes AI's that are perfect Chess players.

• January 28, 2020 at 9:19 pm

It would be interesting to apply this toward cosmology. We have a fairly good understand of the state of the universe as it exists today. With this program, you could input parameters for the Big Bang and inflation to see which original conditions most accurately describe what the Universe is like today.

• January 28, 2020 at 9:25 pm

This is fantastic. Please create lots of elastic gui objects, give them a football and make goals their goal.

• January 28, 2020 at 9:31 pm

i hope this gets used a ton for cgi in movies, like how you said the smoke bunny can be set up so it looks like it could actually happen, that would be cool in movies/art

• January 28, 2020 at 9:39 pm

😱

• January 28, 2020 at 9:40 pm

could this be used to teach robots to walk?

• January 28, 2020 at 9:43 pm

Extraordinary! Thanks for the excellent videos

• January 28, 2020 at 9:56 pm

I'll be impressed when they do it irl and not a simulation.

• January 28, 2020 at 9:58 pm

Excellent news! Thank you very much for sharing!

• January 28, 2020 at 9:58 pm

I can think of a use case. I've been wanting to figure out a way to make fluid simulations that loop seamlessly, so that you could bake the results and play them back in a game scene or something, without having to compute the simulation in real time. The problem, and where this would come in, is that it's difficult to get the simulation to naturally end in the same state it starts.

• January 28, 2020 at 9:59 pm

It would be amazing to see this method deployed to look into the self-assembly in soft condensed matter. Interesting physics can be investigated from several important processes such as protein aggregations (pore formers, ion channels, etc) diblock copolymer formation, multiphase behaviour,

• January 28, 2020 at 10:09 pm

Funny enough, a more complicated variation of the billiards problem known as "No Friction Lunar Pool" was solved by Bisqwit et-al back in 2001. I'd provide a link, but Youtube dislikes comments with links.

• January 28, 2020 at 10:25 pm

I'm waiting for more papers on computational linguistics.

• January 28, 2020 at 10:33 pm

I wish I understood this

• January 28, 2020 at 10:57 pm

Heavy Duty & amazing stuff is right. It has only fairly recently dawned on me just how effective & especially how incredibly accessible ML has become.

• January 28, 2020 at 10:59 pm

The gooey object simulation looks similar to the new xenobots that were recently created from skin/heart cells!

• January 28, 2020 at 11:06 pm

wow

• January 28, 2020 at 11:18 pm

Hey I wanted to ask if you could make a video about what those papers are in general about or what this field of expertise is and if there are any videos or if you know of any good videos or books that teach the basics well? I just don't have a clue about rendering or simulations or ai but think it's so interesting. But I feel like I need some foundation to appreciate your videos more.

• January 28, 2020 at 11:36 pm

Highlight of my day

• January 28, 2020 at 11:41 pm

Train a neural network to distinguish fake goldfish from real goldfish

• January 28, 2020 at 11:41 pm

n-body simulation?

• January 28, 2020 at 11:44 pm

Hold on to your papers! What a time to be alive!

• January 29, 2020 at 12:11 am

Could this AI be used to create novel designs? Where a desired output is defined but the AI has to figure out it's own mechanism to achieve it?

• January 29, 2020 at 1:38 am

OMG this could not have come at a better time! well i have just found my bachelor thesis. thank you

• January 29, 2020 at 1:40 am

Ideally we want actors in games which adapt to the player – entertaining, yet directing them along a storyline. This algorithm might be at the heart of approaching this goal?

• January 29, 2020 at 1:52 am

Can that squirrel to goldfish algorithm be used to design a hat or jacket pattern to confuse a facerecognition algorithm to identify me as some odious politician?

• January 29, 2020 at 2:00 am

So are you going to use it to re-do the smoke bunny?! We want to see it!

• January 29, 2020 at 2:13 am

I love your phrase "What a time to be alive". It so is!

• January 29, 2020 at 2:17 am

that's actually crazy… can't wait to see what the future holds.

• January 29, 2020 at 2:20 am

<3

• January 29, 2020 at 2:28 am

You got to the pool balls, and I was like use it for SPAAAAACE!
If this gets accurate enough, and if the physics of it are solid enough, it could be highly useful for estimating trajectories of known objects, such as asteroids in the belt between us and jupiter, to plan flight paths for missions.
Already we have simulators for that of course, but if applied correctly, this could make it easier by a lot.
— Everything you show can help make awesome games, but also much of it could be useful For Science! 😀

• January 29, 2020 at 2:28 am

what font do you use for ur research paper? it looks beautiful.

• January 29, 2020 at 2:37 am

Can anyone else not see the goldfish or is that just me?

• January 29, 2020 at 2:43 am

Reminds me (just a little) of the FFTW. FFT is sped up greatly through the use of mathmatical identities and code transformations.

• January 29, 2020 at 2:55 am

Imagine if you could actually get this to run in a game, that would be amazing!
Hypothetically then it could be used dynamically! Imagine if a character model in a game didn't just perform a preset animation!

A player model, or an NPC could approach an environment based on the physics, meaning that if you give them a lump of rock instead of a climbing animation they could actually try to scale it.

Also those rocks could break if there are weak-points when they're generated and it wouldn't have to be like a planned 'this always breaks when you climb it' story event.

It could mean a voxel-ish sandbox with superbly immersive adventure elements.

Also in games with fighting imagine an AI shifting their posture to more accurately target your characters location based on the key constraints of the physics. (ex: their leg can't stretch, meaning they can miss if the timing is off, but they won't just kick the same way each time instead they'll shift however their skeleton allows a kick.)

There's just so many possibilities, so many! I agree, what a time to be alive!

• January 29, 2020 at 3:10 am

that sounds to me like reinforcement learning method, or am i wrong?

• January 29, 2020 at 3:19 am

The implications of this for all levels of 3D asset creation are insane, but what is really exciting to me is the combination of the Differentiable rendering and physics. It makes me dream of a world in the not too distant future (maybe houdini 23) where i can sculpt or model a basic form, bring in a photo of the thing i am creating. The differentiable rendering does it's thing and then uses a knowledge base of physically based materials. It then assigns the material properties (wood, metal, glass) to the object based on it's Reflectivity etc, and applies the physics properties , and pushes it back to the 3D software. you then have a photogrammetically correct hi res model that has not only the visual properties of the material but the physical properties like mass and density as well. all adjustable and able to be worked on further. And then I can do it all in VR. SIgn me up!
Luv the videos, thanks for keeping me informed! 🙂

• January 29, 2020 at 3:43 am

Useful in systems where you can afford a certain level of imprecision and where you want to limit the number of interventions (energy consumption) necessary to reach an objective.

• January 29, 2020 at 4:15 am

Taking biochemistry now, I've been thinking a lot about proteins, their seemingly endless multitudes of shape and function, and the ways in which the interact with each other and other substrates. Do you know of any people working on a program or AI that helps determine the ways in which they do these tasks?

• January 29, 2020 at 4:33 am

u could do pathfinding for Dwarf Fortress multi-threaded

• January 29, 2020 at 4:53 am

Really like this paper has a useful effect while still being simple and easy to use and understand. One major use I can think of for this is perhaps using it as a way to make an AI able to take far more information in when looking at high FPS and/or high resolution inputs without a lot of shortcuts that could remove information. For example it could be used to know the outcome of a interaction that triggers a set outcome several seconds before a "Normal" AI does thus allowing it to ignore certain parts of the input until after it knows the outcome is resolved. One variable that could be used for this the average time it takes to resolves said outcome. This could drastically increase the speed in which AIs can respond to other inputs and perhaps even have a system where a primary AI uses this method and is less precise due to the removal of what it thinks is "Unnecessary" and a secondary AI which uses more traditional methods that are more precise even if they are slower.

• January 29, 2020 at 4:56 am

Does this solve the three body problem?

• January 29, 2020 at 5:32 am

Use this in a meta way: design the game such that you have an idea of what the outcome of the game should be: how it should play and what the final package of the design should look like, and then use this differentiable system to figure out what mechanics and mechanics-adjustments need to be created to get it just right.

Plausibly this could be applied to currently existing video games like fighting games, and within a strict framework, it would be possible to give the algorithm free reign to end up with a final version that has something like "no character has a better than 55% winrate against any other character".

• January 29, 2020 at 6:01 am

this algorithm is about to rip a hole in our stock market

• January 29, 2020 at 6:09 am

3D Modeler's HATED this AI. Here's why!

• January 29, 2020 at 6:10 am

this whoal video is basically what happens when you turn RTX ON and OFF

• January 29, 2020 at 6:14 am

Wow! Absolutely amazing, what a great way to end the day!

• January 29, 2020 at 6:24 am

The program is made to fool the network when it believed Squirrel as Goldfish right? if it is, then this can be used as another Data Augmentation technique to train the model better.

• January 29, 2020 at 6:43 am

This is scary.

• January 29, 2020 at 6:43 am

all of these are accomplished by one guy 胡渊鸣.

• January 29, 2020 at 6:59 am

Controlling traffic lights to optimize flow would be possible in this?

• January 29, 2020 at 7:00 am

The squirrel still looks like a squirrel to me, am i missing something? Still cool nonetheless.

• January 29, 2020 at 7:04 am

Wait… did you just reverse the flow of entropy????

• January 29, 2020 at 7:13 am

Weather, Military, Medical, and Entertainment.

• January 29, 2020 at 7:15 am

I'd love to see it design the strongest bridge possible with a given amount of material and a variety of forces that could collapse it.

• January 29, 2020 at 7:16 am

3:02 forming Ying-yang is has a similar look to the art technique Ebru paper marbeling. Ebru uses ink suspended on water and tools (like a human hair) are dragged through it to create designs

• January 29, 2020 at 7:20 am

Clever👍

• January 29, 2020 at 7:20 am

3:36 . . . im still seeing 3 squirrels

• January 29, 2020 at 7:28 am

if it can be applied to solve some problems and then be applied to the end code to optimize it as much as possible..
or even applied to optimize itself.. thats just amazing we're getting close to answer to life, the universe and everything

• January 29, 2020 at 7:29 am

This is how A.G.I. will learn to navigate the world faster.

• January 29, 2020 at 7:35 am

Second law of thermodynamics and entropy what again?

• January 29, 2020 at 7:37 am

Impressive. The very first example you showed has quite obvious utility a a tool to help 3d asset creation – the challenge for game makers and fx artists is always to take some concept sketch or photo and create a textured 3d model of it. This tool could massively simplify that process if it could use noisier inputs and export standard formats.

• January 29, 2020 at 7:43 am

It would be useful to train a algorithm like this to be used as a slicer for a 3d printer. All you would need would be a few test prints to calibrate the printer then the filament. Then it could run simulations until it produced a gcode that would always work and always have the properties you set it for. Within reason of course. Then after a few generations and a module on the printer along with a few cameras, as well as some parts. You might be able to get to the point where you can do mold less injection molding.

• January 29, 2020 at 7:43 am

would it be possible to show this algorithm how a human walks and it could then implement it into the movement of a character in a video game?

• January 29, 2020 at 8:07 am

please explain what differentiable physics means in this context – I still have no clue what is differentiable here. Lovely pictures though…

• January 29, 2020 at 8:21 am

The jello is few lines of code because it’s written using language and framework that researchers developed. A classified in PyTorch is few lines of code for the similar reason.

• January 29, 2020 at 8:22 am

Watching these videos makes me want to get into research…

Need to do my masters first at least I guess

• January 29, 2020 at 8:30 am

It's so weird to hear how hard it is but then few minutes later hear that some dude just did a thing with some short code. O_o

• January 29, 2020 at 8:31 am

Nice paper! Just wanted to point out that it seems to work well on continuous systems, the billiard balls are setup close together and function like a continuous domain, the possible pairwise collisions are small because they're so close together. For example if billiard balls where more spread apart and one had to do a particular sequence of collisions to get some ball in the pocket, I don't think this technique would scale since there would be a lot more possibilities of collisions to consider. Anyway, nice paper, and nice video!

• January 29, 2020 at 8:58 am

I didnt understand that water ripple thing, am i stupid?

• January 29, 2020 at 9:05 am

I'm not a scholar but enjoy your videos.

• January 29, 2020 at 9:08 am

You have to be very intelligent to be able to read and interpret some of these papers!

• January 29, 2020 at 9:21 am

emergent complexity Pog

• January 29, 2020 at 9:25 am

Wow

• January 29, 2020 at 9:38 am

Principally we can use the knowledge of this processing dynamic to advance conscientious creative freedom. I Praise your conduct!

• January 29, 2020 at 9:44 am

By differentiable you mean something that's in calculus?

• January 29, 2020 at 9:57 am

I don't get it.

• January 29, 2020 at 10:03 am

could it be used for photorealism of both life and comic book images at the same time to produce effects and subtle movement similar to what was used in "Spider-Man into the spider-verse"? matching 2d animated characters (or possibly cel/ toon shaded 3d character animatons) to many real models of an actor to produce near perfect animation? like matching sets of poorly drawn animation to a video of a real person running, and have it be "fixed"?

• January 29, 2020 at 10:10 am

So this could be used to generate 3D models/textures from photos significantly cutting down production time/costs.

• January 29, 2020 at 10:20 am

This episode has been supported by waits, and by asses

• January 29, 2020 at 10:20 am

or even better

• January 29, 2020 at 10:21 am

So, this can be used to achieve from a similar input the desired output? I'm really curious on how it would handle encryption – instead of a pear – show it encrypted text – and it's output

then re-do for another encrypted text – with the same key (yes I know – even the keys have entropy in it – so it's never the same – but still) – then with a different key

for some years now I have wondering how ML can help break ciphers

• January 29, 2020 at 10:22 am

I bet the goldfish is actually the blob of leafs behind the squirrel. (Directly to right from squirrel's arms) Squirrel was detected only 42% originally and ripples did clearly distort its ears etc to hide it more.

• January 29, 2020 at 10:24 am

Thank you for simplifying this!

• January 29, 2020 at 10:48 am

Creating adversarial images for neural networks using fluid simulation sure isn't what i expected to see today, but I'm into it.

• January 29, 2020 at 10:56 am

Reality is whatever I want to to be.

Thanos would be proud.

• January 29, 2020 at 11:02 am

Everytime Impressed!

• January 29, 2020 at 11:04 am

Host : What a time to be alive.

Me: I do agree

Coronavírus: I am about to end those men's entire career

• January 29, 2020 at 11:04 am

Push the envelope, watch it bend.

• January 29, 2020 at 11:11 am

CIA: DiffTaichi, we need to naturalize a target with just a paperclip and a rubber band.

• January 29, 2020 at 11:25 am

Use it to develop an experiment that would defeat Maxwell's demon and reverse entropy. Free energy for everyone 🙂

• January 29, 2020 at 11:27 am

I want to live in a world where this page gets as many views as pewdiepie.

• January 29, 2020 at 11:31 am

Amazing stuff. Audio is very bassy in this vid.

• January 29, 2020 at 11:48 am

Very cool, but… this was known several decades ago as "optimal control" or "differential programming": to steer a general dynamical system from an initial state to a final state, so that the system's trajectory is optimal in some user defined sense. The emphasis of the paper seems to be on the efficient implementation of the algorithm, especially the automatic differentiation of the physical model. Many languages allow this, but apparently their implementation is more efficient.

If you find these results stunning, check out what it is done for the Global Trajectory Optimisation Competition (GTOC), or any complex trajectory optimisation 😉

• January 29, 2020 at 11:57 am

I don't see what you mean, there is only a goldfish