Strange...I have a six month old and baby sees my in-laws minimum 3 times per week for about an hour each time. They live a 3 minute walk away.
Other weeks, my husband goes over twice to their house for 20 minutes while I eat lunch and then Saturday we'll be at their house for a couple hours for lunch.
I was, however, quite resentful of feeling like I owed them visits. There was a period in which they were seeing the baby 5 times per week and it felt like it was never enough. My husband's mother would guilt trip us "I haven't seen you in two days" or "finally I get to see you" or "I didn't see the baby yesterday".. it was annoying me that I felt like I was spending my maternity leave arranging visits for them. They were too pushy and have since backed off as I've been a lot more assertive. Read my loong threads on "MIL drops by unannounced". I have two threads.
Do NOT go there without calling or be too pushy. But three weeks is far too long to not see the baby.
What was your relationship like before baby came??? Mine was stellar! It's only after having baby that I ran into problems.
Will you be a caregiver to this baby? You can hopefully offer childcare further along and be more involved.
I suggest cooking something delicious and asking if you can bring it over.
They don't realize they are hurting you, trust me! Unless you had a bad relationship before, DIL is not doing this on purpose.
I would just text her something like "Hi (name), I hope you are feeling better. I know this is an exhausting time. We're here for you when you need us. I've made (dish) because I know you like it. Let me know when we can drop by for a quick visit to drop it off."
Or don't cook anything and just ask when you can drop by to see THEM, not the baby.
When you do get access and take care of the kid, I recommend getting a few "ground rules" to know some of DILs and son's preferences. This can squash tension because what DIL can say to her mom "stop that, mom!", she can't say freely to you.