bug: Cannot fetch dead actor for respawning

[CristianoBarone]

Bug description
When killing an actor, if the error "actor is not alive" is thrown for a child of said actor, the actor won't be respawnable

How to reproduce it?

exists, err := r.self.ActorSystem().ActorExists(ctx, id) //exists and err are true and nil
if err == nil {
	if exists {	
		_, pid, err := r.self.ActorSystem().ActorOf(ctx, id) //pid and err are both nil
		if err != nil {
			fmt.Printf("Failed to fetch respawnable actor: %v\n", err)
			return &Error{Message: "Unable"}
		} else {
			if !pid.IsRunning() { //fails because pid is nil
				pid.Reinstate(pid)
			}
		}
	}
} else {
	fmt.Printf("Failed to lookup actor: %v\n", err) //never printed
	return &Error{Message: "Unable"}
}
err = r.self.ActorSystem().SpawnOn(ctx, id, NewRule(), goakt.WithLongLived()) 
if err != nil {
	fmt.Printf("Failed to create actor: %v", err)
	return &Error{Message: "Unable"}
}
return &Error{Message: "Done"}

Expected behavior
Dead actor is found and respawned.

Screenshots
.

Library Version:

• Go-Akt version: 3.9.1
• Go version: 1.24

Additional context
.

[Tochemey]

@CristianoBarone when an actor stop existing, it is completely wiped out from the system and there is no way to respawn them except creating a new instance. That is the lifecycle of an actor. It is only Grains that can get reactivated on message.

I would like to clarify that first before looking into your issue.

[Tochemey]

Bug description When killing an actor, if the error "actor is not alive" is thrown for a child of said actor, the actor won't be respawnable

How to reproduce it?

exists, err := r.self.ActorSystem().ActorExists(ctx, id) //exists and err are true and nil
if err == nil {
if exists {
_, pid, err := r.self.ActorSystem().ActorOf(ctx, id) //pid and err are both nil
if err != nil {
fmt.Printf("Failed to fetch respawnable actor: %v\n", err)
return &Error{Message: "Unable"}
} else {
if !pid.IsRunning() { //fails because pid is nil
pid.Reinstate(pid)
}
}
}
} else {
fmt.Printf("Failed to lookup actor: %v\n", err) //never printed
return &Error{Message: "Unable"}
}
err = r.self.ActorSystem().SpawnOn(ctx, id, NewRule(), goakt.WithLongLived())
if err != nil {
fmt.Printf("Failed to create actor: %v", err)
return &Error{Message: "Unable"}
}
return &Error{Message: "Done"}
Expected behavior Dead actor is found and respawned.

Screenshots .

Library Version:

• Go-Akt version: 3.9.1
• Go version: 1.24

Additional context .

• You only use Reinstate when an actor is suspended.
• If the actor exists as your code stated then it can be a local actor or a remote actor:
  • Local actor: ActorOf return not nil PID
  • Remote actor: ActorOf return nil for PID and an address to use.

Therefore if the actor exist you just send it message using SendSync or SendAsync. From your code this actor is a remote actor because the PID is nil.

I am of the opinion this is an issue related to how you are using the library.

[CristianoBarone]

@CristianoBarone when an actor stop existing, it is completely wiped out from the system and there is no way to respawn them except creating a new instance. That is the lifecycle of an actor. It is only Grains that can get reactivated on message.

I would like to clarify that first before looking into your issue.

Then the problem is that when the actor is stopped it isn't wiped from the system, so it cannot be spawned again. I will send you the code that takes care of stopping the actors tomorrow.

[Tochemey]

@CristianoBarone Dead actor do not exist in the system. I have provided a sample code here for you to peruse and run it locally:

package main

import (
	"context"
	"errors"
	"fmt"
	"os"
	"time"

	"github.com/tochemey/goakt/v3/actor"
	gerrors "github.com/tochemey/goakt/v3/errors"
	"github.com/tochemey/goakt/v3/goaktpb"
	"github.com/tochemey/goakt/v3/log"
)

func main() {
	ctx := context.Background()
	logger := log.DiscardLogger

	fmt.Println("Starting actor system...")
	actorSystem, _ := actor.NewActorSystem("Test", actor.WithLogger(logger))
	if err := actorSystem.Start(ctx); err != nil {
		fmt.Printf("failed to start actor system: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("\nSpawning actor...")
	pid, err := actorSystem.Spawn(ctx, "actor1", &MyActor{}, actor.WithLongLived())
	if err != nil {
		fmt.Printf("failed to spawn actor: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("\nStopping actor...")
	if err := pid.Shutdown(ctx); err != nil {
		fmt.Printf("failed to stop actor: %v\n", err)
		os.Exit(1)
	}

	// wait for a while to ensure the actor is stopped
	time.Sleep(time.Second)

	ok, err := actorSystem.ActorExists(ctx, "actor1")
	if err != nil {
		fmt.Println("failed to check if actor exists:", err)
		os.Exit(1)
	}

	if ok {
		fmt.Println("actor should be stopped but still exists")
		os.Exit(1)
	}

	addr, pid, err := actorSystem.ActorOf(ctx, "actor1")
	if err != nil {
		if !errors.Is(err, gerrors.ErrActorNotFound) {
			fmt.Println("failed to get actor:", err)
			os.Exit(1)
		}
	}

	if addr != nil || pid != nil {
		fmt.Println("actor should be stopped but still exists")
		os.Exit(1)
	}

	fmt.Println("Actor stopped successfully")

	fmt.Println("\nStopping actor system...")
	_ = actorSystem.Stop(ctx)
}

type MyActor struct{}

var _ actor.Actor = (*MyActor)(nil)

func (x *MyActor) PreStart(*actor.Context) error {
	fmt.Println("PreStart")
	return nil
}

func (x *MyActor) Receive(ctx *actor.ReceiveContext) {
	switch ctx.Message().(type) {
	case *goaktpb.PostStart:
		fmt.Println("PostStart")
	default:
		ctx.Unhandled()
	}
}

func (x *MyActor) PostStop(*actor.Context) error {
	fmt.Println("PostStop")
	return nil
}

[Tochemey]

@CristianoBarone can you please elaborate on this:

When killing an actor, if the error "actor is not alive" is thrown for a child of said actor, the actor won't be respawnable

[CristianoBarone]

I have a system in which an actor with multiple children is spawned, at a point the actor must be terminated. If for any reason one of the children is not alive, the actor will throw that error and fail to terminate, however it will terminate at the second try.

When at a latter moment in time the actor is to be respawned, it won't, stating the actor already exists.

[Tochemey]

That isn’t a GoAkt bug — it’s related to your application’s design. Unfortunately, I can’t provide much direct help with that. You may need to revisit your design with respect to the actor model paradigm.

I hope you read this part of the doc:https://tochemey.gitbook.io/goakt/features/supervision.

[Tochemey]

@CristianoBarone let us continue the conversation here. If it is a real issue then I can move it back to issues. Thanks

[Tochemey]

That will be great.

[CristianoBarone]

Hi, I have ran the code you provided and it works, I've also reviewed the documentation and I am able to spawn previously killed actors.

One issue persists; When an actor is killed the following message is given:
parent=(MyParentId) failed to disown descendant=(ChildActorId): actor is not alive

Trying to kill it a second time does the trick. Why is this? Can you please elaborate on why this happens and which error struct is that? The actor isn't killed throughout the parent's life and this only happens when shutting down.

If anything, can you implement a feature to forcefully shutdown an actor by skipping trivial errors like this? Thank you for your patience.

[Tochemey]

Ok I will take a look. I have an idea. Thanks

[Tochemey]

Hi, I have ran the code you provided and it works, I've also reviewed the documentation and I am able to spawn previously killed actors.

One issue persists; When an actor is killed the following message is given: parent=(MyParentId) failed to disown descendant=(ChildActorId): actor is not alive

This happens when the actor is already dead. That means some code in your application kills the actor. I have a code path to ensure killed actors do not come into the picture of shutdown process. Remember everything in GoAkt is reactive which means timing issue can occurs even though the library is robust enough to cater for this.

Trying to kill it a second time does the trick. Why is this? Can you please elaborate on why this happens and which error struct is that? The actor isn't killed throughout the parent's life and this only happens when shutting down.

I have no clue on this process. All I can say that when killing an actor all its descendants will also be killed. It is by design.

If anything, can you implement a feature to forcefully shutdown an actor by skipping trivial errors like this? Thank you for your patience.

I don't think we need such a feature.

[CristianoBarone]

This happens when the actor is already dead. That means some code in your application kills the actor.

No code does this at the moment, but I did fix the problem by killing all children before shutting down the actor. I think there may be a race condition at play here.

[Tochemey]

@CristianoBarone would you mind sharing your experience with GoAkt if you guys are using it in production?

[CristianoBarone]

I'm currently assigned to a project which is not yet running in production, however, GoAkt has already proven very fast and most importantly easy to use since I've hardly ever worked using an actor model. Of course I can't speak on behalf of my employer but surely there's a reason why we're using GoAkt instead of other libraries.

One problem I have to point out though, is the massive amount of messages outputted, it would be easier if there was a way to disable logging per-category of message because sometimes the volume of console output makes debugging harder other than possibly slowing down the whole system, although GoAkt is very fast.

[Tochemey]

@CristianoBarone there is a logger interface that anyone can implement with any custom settings. I believe that should suffice.

[Tochemey]

@CristianoBarone I released a new tag that should get some of your concerns addressed except for the logging :) https://github.com/Tochemey/goakt/releases/tag/v3.9.2

[CristianoBarone]

Yeah I saw it, will try tomorrow morning

[CristianoBarone]

@Tochemey The new release worked perfectly, now killing a parent kills all children without margin of error, before I had to cycle through them and kill them one by one to have a constant result.

[Tochemey]

@CristianoBarone That is good to know. I will close the discussion. Remember I do also provide consultancy in actor model system design. In case you and your team need some sort of support you can always reach out.