CAMEL-22292: Add --infra option to camel cmd send for infrastructure services

[pkalsi97]

Description

This PR adds a feature to the camel cmd send command to support sending messages directly to infrastructure services (like NATS, Kafka, Redis, etc.) that are started with camel infra run and updated docs.

Previously, users had to manually specify server connection details, but now the command automatically reads connection information from JSON files created by infrastructure services.

How it works:

1. When --infra option is specified, we locate the corresponding infra service JSON file.
2. Read connection details form the JSON file.
3. If endpoint is specified, construct the full endpoint URI
4. Use existing message sending mechanism to send the message to the infrastructure service
5. Handle cases where infrastructure services don't exist with appropriate error messages

Target

• I checked that the commit is targeting the correct branch (Camel 4 uses the main branch)

Tracking

• If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

• I checked that each commit in the pull request has a meaningful subject line and body.

• I have run mvn clean install -DskipTests locally from root folder and I have committed all auto-generated changes.

[Croway]

Hi @pkalsi97 thanks for the PR, this is really interesting. By any chance did you test all the infrastructure? Some time ago I did an alignment of the infra properties (the key you can find in the json), with the component's properties, therefore, the mapping should work automatically, it should be safe to use jsons properties as key/values in the command. But I am not 100% sure it will work for all the usecases.

[davsclaus]

You need to use camel-catalog it has an API for building endpoint URIs with

uri = catalog.asEndpointUri("netty-http", map, true);

Where map is a map with key/value pairs.

This will build the uri correct according to the given component.

[pkalsi97]

Hey ! @Croway @davsclaus

First of all apologies, i could have tackled this issue better by first discussing the approach and solution in the issue itself before lodging the PR. I got way to excited haha. Thank a lot for your inputs i'll incorporate these in the followup.

Regarding the --infra option coverage, this got way to overlooked in the PR. I am fairly determined to cover all services, what testing approach is expected? Will Unit test (Mock the JSON for each service -> Verify the generated endpoint URI) will that be sufficient?

[Croway]

@pkalsi97 no worries! moreover, I like the idea

yeah, mocking the JSON sounds good, this way we won't have to to download all the images on the CI.

We have to make sure that all the properties in the test-infra services like https://github.com/apache/camel/blob/main/test-infra/camel-test-infra-kafka/src/main/java/org/apache/camel/test/infra/kafka/services/KafkaInfraService.java are either supported by the component as is (brokers in this case), or somehow handled, in this case getBootstrapServers is not a supported property by the component. I deprecated some of those in the past, but as you can see from the Kafka, I didn't deprecate all of them :), brokers and getBootstrapServers contain the same value at the moment.

[davsclaus]

We can remove the deprecated options so this tool wont see "bad" options.
Then just add a note in the 4.18 upgrade guide.
Its okay, as this feature is powerful if we can get this to work correctly

[pkalsi97]

@Croway and @davsclaus Thanks a lot for the inputs.

I have implemented some refinements based on the feedback

1. Now using catalog.asEndpointUri(scheme, properties, true) as suggested to build endpoint URIs.
2. hardcoding a servers property is removed and now we uses service-specific property mappings for components requiring special handling
3. Adds a unit test. All infra services are covered. tests are built against the actual *InfraService interface method names from test-infra modules

[pkalsi97]

After a bit more digging it looks like not all cases in the test are setup correctly, i'll work on fixing it. I should have manually run every infra to validate my test cases with the actual. In some infra services turn out the test is validating if the properties exist but did not validate that the Camel component actually accepts those properties.

[Croway]

ok, instead of having this logic here, can you update the *InfraService in the test-infra? for example, the in the MosquittoInfraService you can add the brokerUrl so that we can get rid of this switch.

[pkalsi97]

sure, will do. I have decided to change my approach a bit. Not able to come to a concrete solution because i was trying to avoid manually running the actual infra and validating if the command correctly works. i'll spend some time setting up a script locally to validate everything first then alight the unit test to that.

[pkalsi97]

Would it be acceptable to include an integration test in this PR that uses Testcontainers to spin up real services and actually test the working of the command ? We can skip it by default in the CI but it can be run locally during development / making changes. What do you think ?

[Croway]

@pkalsi97 for example I've added brokerUrl to MosquittoInfraService and implemented it like

@Override
    public String brokerUrl() {
        return "tcp://localhost:" + container.getMappedPort(CONTAINER_PORT);
    }

in MosquittoLocalContainerInfraService
and

@Override
    public String brokerUrl() {
        return System.getProperty(MosquittoProperties.BROKER_URL);
    }

in MosquittoRemoteInfraService

this way, the supported property is part of the output json when running camel infra run mosquitto

{
  "brokerUrl" : "tcp://localhost:1883",
  "getPort" : 1883
}

Now we can get rid of buildPropertiesMap with something like:

Map<String, String> properties = new HashMap<>();
        catalog.componentModel(scheme).getEndpointOptions().stream()
                .map(ComponentModel.EndpointOptionModel::getName)
                .filter(name -> connectionDetails.containsKey(name))
                .forEach(name -> properties.put(name, connectionDetails.getString(name)));

This way, only the supported properties are added to the endpoint properties.

We should aim to adapt the *InfraService this way even for users will be clear how to use the json output.

I hope it is clear, let me know if you have some doubts

[pkalsi97]

Yes this makes it clear.l, thanks!

[Croway]

Would it be acceptable to include an integration test in this PR that uses Testcontainers to spin up real services and actually test the working of the command ? We can skip it by default in the CI but it can be run locally during development / making changes. What do you think ?

yeah, there is already an integration test for the infra that uses FTP, FTP is the only test-infra service that does not use testcontainers, you can use that for integration test