SystemVerilog Fork Disable "Gotchas"

This is a long post with a lot of SystemVerilog code. The purpose of this entry is to hopefully save you from beating your head against the wall trying to figure out some of the subtleties of SystemVerilog processes (basically, threads). Subtleties like these are commonly referred to in the industry as "Gotchas" which makes them sound so playful and fun, but they really aren't either.

I encourage you to run these examples with your simulator (if you have access to one) so that a) you can see the results first hand and better internalize what's going on, and b) you can tell me in the comments if this code works fine for you and I'll know I should go complain to my simulator vendor.

OK, I'll start with a warm-up that everyone who writes any Verilog or SystemVerilog at all should be aware of, tasks are static by default. If you do this:

module top;
   task do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join 
   end 
endmodule 

both do_stuff calls will wait for 5 time units, and you see this:

waited 5, then did stuff
waited 5, then did stuff

I suppose being static by default is a performance/memory-use optimization, but it's guaranteed to trip up programmers who started with different languages. The fix is to make the task "automatic" instead of static:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join 
   end 
endmodule 

And now you get what you expected:

waited 5, then did stuff
waited 10, then did stuff

Now let's get a little more tricky. Say you want to only wait until one of the forked processes finishes, so you do this:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
   end 
endmodule 

You'll get this output:

waited 5, then did stuff
fork has been joined
waited 10, then did stuff

That's fine, but that extra action from the slower do_stuff after the fork-join_any block has finished might not be what you wanted. You can name the fork block and disable it to take care of that, like so:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork : do_stuff_fork
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
      disable do_stuff_fork;
   end 
endmodule 

Unless your simulator, like mine, "in the current release" will not disable sub-processes created by a fork-join_any statement. Bummer. It's OK, though, because SystemVerilog provides a disable fork statement that disables all active threads of a calling process (if that description doesn't already make you nervous, just wait). Simply do this:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork : do_stuff_fork
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
      disable fork;
   end 
endmodule 

And you get:

waited 5, then did stuff
fork has been joined

Nothing wrong there. Now let's say you have a class that is monitoring a bus. Using a classes are cool because if you have two buses you can create two instances of your monitor class, one for each bus. We can expand our code example to approximate this scenario, like so:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable do_stuff_fork;
   endtask 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule 

Note that I went back to disabling the fork by name instead of using the disable fork statement. This is to illustrate another gotcha. That disable call will disable both instances of the fork, monitor 1's instance and monitor 2's. You get this output:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 fork has been joined
main fork has been joined

Because disabling by name is such a blunt instrument, poor monitor 2 never got a chance. Now, if you turn the disable into a disable fork, like so:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule 

You get what you expect:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 waited 7, then did stuff
monitor 2 fork has been joined
main fork has been joined

It turns out that, like when you disable something by name, disable fork is a pretty blunt tool also. Remember my ominous parenthetical "just wait" above? Here it comes. Try adding another fork like this (look for the fork_something task call):

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   function void fork_something();
      fork 
         # 300 $display("monitor %0d: you'll never see this", id);
      join_none 
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork_something();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
 
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule 

The output you get is:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 waited 7, then did stuff
monitor 2 fork has been joined
main fork has been joined

Yup, fork_something's fork got disabled too. How do you disable only the processes inside the fork you want? You have to wrap your fork-join_any inside of a fork-join, of course. That makes sure that there aren't any other peers or child processes for disable fork to hit. Here's the zoomed in view of that (UPDATE: added missing begin...end for outer fork):

task monitor();
   fork_something();
   fork begin 
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   end 
   join 
endtask 

And now you get what you expect:

monitor 2 fork has been joined
monitor 1 fork has been joined
monitor 1 waited 6, then did stuff
monitor 2 waited 7, then did stuff
main fork has been joined
monitor 1 waited 11, then did stuff
monitor 2 waited 12, then did stuff
monitor 2: you'll never see this
monitor 1: you'll never see this

So, wrap your fork-join_any inside a fork-join or else it's, "Gotcha!!!" (I can almost picture the SystemVerilog language designers saying that out loud, with maniacal expressions on their faces).

But wait, I discovered something even weirder. Instead of making that wrapper fork, you can just move the fork_something() call after the disable fork call and then it doesn't get disabled (you actually see the "you'll never see this" message, try it). So, you might think, just reordering your fork and disable fork calls and that will fix your problem. It will, unless (I learned by sad experience) the monitor task is being repeatedly called inside a forever loop. Here's a simplification of the code that really inspired me to write this all up:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   function void fork_something();
      fork 
         # 30 $display("monitor %0d: you'll never see this", id);
      join_none 
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask // do_stuff 
 
   task monitor_subtask();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
      fork_something();
   endtask 
 
   task monitor();
      forever begin 
         monitor_subtask();
      end 
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
 
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join_none 
      $display("main fork has been joined");
      # 60 $finish;
   end 
endmodule 

The fork inside the fork_something task will get disabled before it can do its job, even though it's after the disable fork statement. Gotcha!!! Muhahahah!

My advice? Just always wrap any disable fork calls inside a fork-join.

Comments

Dave Rich said…
Hi Bryan, Nice set of examples to capture the gotchas.

A key concept to remember is that the call stack and process hierarchy are related, but orthogonal concepts. It does create some weird interactions, but the resulting behavior of wait/disable fork is straightforward if you can keep the concepts separate.

BTW, the reason tasks/functions are static by default outside of class methods has to do with backward compatibility with legacy Verilog. As a pure Hardware Description Language, it had no concept of dynamic or automatically allocated storage.
Unknown said…
Hi Bryen. your post was much of a help to me (after banging my head against the wall). i t works in most places, but i have one place that i have a fork of an agent that disabled other fork of my Reference Model,. i wrapped the agent's fork and it still disables the RM's fork. (to make sure that's the reason my fork doesn't even start, i removed the agent's fork and its all working fine.) do you have a solution for that?

10X
Chaggster
Bryan said…
Chagay,

Sorry I'm so late to reply, this month has been really busy for me at home and at work. I'm not sure what your problem might be. It'd be easier if I could see some code. Do you think you could come up with a simplified example? Posting it on http://www.edaplayground.com/ would be really helpful as we could see how it behaves with different simulators.
almkglor said…
Hi Bryan,

The real reason why the last example fails can be seen if we UNROLL the forever loop a bit.

First, let's factor in monitor_subtask() into monitor():

forever begin
fork : do_stuff_fork
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
fork_something();
end


Now let's unroll the loop once:


begin
fork : do_stuff_fork1
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
fork_something();
forever begin
fork : do_stuff_fork
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
fork_something();
end
end


So far so good.

Then let's unroll it another time:


begin
fork : do_stuff_fork1
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
/**/
fork_something();
fork : do_stuff_fork2
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
/**/
fork_something();
forever begin
fork : do_stuff_fork
do_stuff(id+5);
do_stuff(id+10);
join_any
disable fork;
fork_something();
end
end



Notice I added "/**/" comments. As you can see, when unrolled, the fork_something() call comes *BEFORE* the disable fork statement.

What we really need is thread id's and a way to kill and wait on threads using their id's, not a disable-fork sledgehammer.
Anonymous said…
Very good one with great examples
Anonymous said…
Sorry, late to the party, but great explanation!

I may make a `define that does the fork\nfork and join_any\ndisable fork;\njoin

Popular posts from this blog

'git revert' Is Not Equivalent To 'svn revert'

Put /etc Under Revision Control (with git)