SpringCloud AWS etiketine sahip kayıtlar gösteriliyor. Tüm kayıtları göster
SpringCloud AWS etiketine sahip kayıtlar gösteriliyor. Tüm kayıtları göster

1 Ağustos 2023 Salı

SpringCloud AWS Systems Manager - AWS Systems Manager'dan Bilgileri Alır

Giriş
SpringCloud AWS Secrets Manager kullanımına çok benziyor

Maven
Şu satırı dahil ederiz
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.awspring.cloud</groupId>
      <artifactId>spring-cloud-aws-dependencies</artifactId>
      <version>3.0.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependency>
  <groupId>io.awspring.cloud</groupId>
  <artifactId>spring-cloud-aws-starter-parameter-store</artifactId>
</dependency>
IAM Permissions
Açıklaması şöyle
For access, IAM permissions can be set up. The required Spring Cloud AWS permission is: ssm:GetParameters
Şöyle yaparız
Sample IAM policy granting access to Parameter Store:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:GetParametersByPath",
            "Resource": "*"
        }
    ]
}
application.yaml Dosyası
Açıklaması şöyle
spring.config.import property is used to fetch parameters from AWS Parameter Store and add them to Spring’s environment properties.
Örnek
Şöyle yaparız
spring:
  profiles:
    active: dev
  application:
    name: my-demo-boot
  # AWS parameter store configuration
  cloud:
    aws:
      credentials:
        access-key: <your-access-key>
        secret-key: <your-secret-key>
        profile:
          name: default
      region:
        static: us-east-2
  config:
    import:
      - aws-parameterstore:/config/application_${spring.profiles.active}/

# actuator configuration
management:
  endpoints:
    enabled-by-default: false
    web:
      exposure:
        include: 'health, env'
  endpoint:
    health:
      enabled: true
      show-details: always
    env:
      enabled: true

9 Şubat 2023 Perşembe

SpringCloud AWS S3

Maven
Şöyle yaparız
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>3.0.1</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
Sonra şöyle yaparız
<dependencies>
  <dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-starter-s3</artifactId>
  </dependency>
</dependencies>
application.properties
Açıklaması şöyle
There are two ways to communicate with the S3 bucket.

1. Path style URLs.
2. Virtual Hosted style URLs.

The path style URLs were the older mechanism which used the format https://s3.region-code.amazonaws.com/bucket-name.

While the virtual hosted URL uses the format https://bucket-name.s3.region-code.amazonaws.com

The path style URLs will soon be deprecated and LocalStack already supports the virtual hosted style URLs by default.
Örnek
Şöyle yaparız.
spring:
  cloud:
    aws:
      s3:
        endpoint: http://s3.localhost.localstack.cloud:4566
        region: eu-central-1
      credentials:
        access-key: none
        secret-key: none
      region:
        static: eu-central-1
Örnek
Şöyle yaparız. Bizim için bir tane AmazonS3  bean yaratılır
spring.cloud.aws.credentials.access-key=dummy
spring.cloud.aws.credentials.secret-key=dummy
spring.cloud.aws.s3.region=eu-west-1
Örnek
Şöyle yaparız
@RestController
public class WebController {

  @Value("s3://mybucket/samplefile.txt")
  private Resource s3SampleFile;

  @GetMapping("/data")
  public ResponseEntity<String> getData() {
    try {
      return ResponseEntity.ok(s3SampleFile.getContentAsString(StandardCharsets.UTF_8));
    } catch (Exception e) {
      return ResponseEntity.internalServerError()
                    .body("Could not fetch content");
    }
  }

  @PostMapping("/data")
  public ResponseEntity<String> putData(@RequestBody String data) throws IOException {
    try (OutputStream outputStream = ((S3Resource) s3SampleFile).getOutputStream()) {
      outputStream.write(data.getBytes(StandardCharsets.UTF_8));
    }
    return ResponseEntity.ok(data);
  }
}
AmazonS3  Sınıfı
putObject metodu
Örnek
Şöyle yaparız
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ObjectMetadata;

@Service
@AllArgsConstructor
public class UploadService {

  private final AmazonS3 s3Client;

  public String uploadFile(InputStream file, String fileName) {
    String bucketName = ...;
    s3Client.putObject(bucketName, fileName, file, new ObjectMetadata());
    return s3Client.getUrl(bucketName, fileName).getPath();
  }
}
Unit Test
LocalStackContainer kullanılır

Örnek
Şöyle yaparız
@Testcontainers
@SpringBootTest
@AutoConfigureMockMvc
class SpringBootWithS3ApplicationTests {

  @Autowired
  private MockMvc mockMvc;

  @Value("s3://mybucket/samplefile.txt")
  private Resource s3SampleFile;

  @Container
  private static LocalStackContainer localStackContainer =
    new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
    // to create secrets on startup
    .withCopyFileToContainer(MountableFile.forClasspathResource("script.sh", 0775),
                             "/etc/localstack/init/ready.d/")
    .withServices(LocalStackContainer.Service.S3);

  @BeforeAll
  static void beforeAll() throws IOException, InterruptedException {
    System.setProperty("spring.cloud.aws.s3.endpoint",
      "http://s3.localhost.localstack.cloud:" + localStackContainer.getMappedPort(4566));
    System.setProperty("spring.cloud.aws.s3.region", "eu-central-1");
    System.setProperty("spring.cloud.aws.credentials.access-key", "none");
    System.setProperty("spring.cloud.aws.credentials.secret-key", "none");
  }
 ...
}




27 Aralık 2022 Salı

SpringCloud AWS SQS @SqsListener Anotasyonu

Giriş
Şu satırı dahil ederiz
import org.springframework.cloud.aws.messaging.listener.annotation.SqsListener;
Açıklaması şöyle
As we will use Spring to build the microservice we could use Spring Cloud AWS. It is very easy to build an SQS listener using this library. With little to no configurations, one can create the listener method just by annotation it with @SqsListener annotation.

But this library is limited. The maximum amount of messages that can be fetched at once cant be more than 10. Although it allows parallel processing, receiving up to 10 messages doesn't allow saving messages at once to the database.

Because it receives messages, creates multiple parallel threads, and each thread calls the listener method passing one message as an argument.

deletionPolicy Alanı
Şöyle yaparız
import com.aws.sqs.model.Message; import org.springframework.cloud.aws.messaging.listener.SqsMessageDeletionPolicy;
import org.springframework.cloud.aws.messaging.listener.annotation.SqsListener;

@RestController
@RequestMapping(value = "/sqs")
public class SqsController {
  //queue name
  private static final String QUEUE = "my-queue";
...
  // @SqsListener listens to the message from the queue.
  @SqsListener(value = QUEUE, deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
  public void getMessageFromSqs(Message message, @Header("MessageId") String messageId) {
    logger.info("Received message= {} with messageId= {}", message.toString(), messageId);
  }
}

30 Ağustos 2022 Salı

SpringCloud AWS Secrets Manager - AWS Secrets Manager'dan Bilgileri Alır

Giriş
AWS secret değerlerini actuator ile env altında görebiliriz.
http://localhost:8080/actuator/env
SpringCloud AWS Systems Manager kullanımına çok benziyor

Maven
Şu satırı dahil ederiz
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.awspring.cloud</groupId>
      <artifactId>spring-cloud-aws-dependencies</artifactId>
      <version>3.0.0-M2</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
</dependencies>

<dependency>
  <groupId>io.awspring.cloud</groupId>
  <artifactId>spring-cloud-aws-starter-secrets-manager</artifactId>
</dependency>
Gradle
Şu satırı dahil ederiz
dependencies {
  implementation 'org.springframework.boot:spring-boot-starter'
  implementation('io.awspring.cloud:spring-cloud-starter-aws-secrets-manager-config:2.4.2')
}
application.yaml Dosyası
spring.config.import AWS'ten parametreleri çeker ve bunları Spring’in ortam değişkenlerine (environment properties) ekler

Örnek
Maven ile şöyle yaparız
<dependency>
     <groupId>com.amazonaws</groupId>
     <artifactId>aws-java-sdk-secretsmanager</artifactId>
  </dependency>
  <dependency>
     <groupId>com.amazonaws.secretsmanager</groupId>
     <artifactId>aws-secretsmanager-jdbc</artifactId>
     <version>1.0.5</version>
  </dependency>
application.properties ile şöyle yaparız
spring.datasource.url = (name/path of your secrets manager)
spring.datasource.username = (name/path of your secrets manager)
spring.datasource.driver-class-name = com.amazonaws.secretsmanager.sql.AWSSecretsManagerMySQLDriver
Örnek - Other Secret Type

Örnek
yaml dosyası şöyle olsun. 3 tane farklı secret dosyası içinde sev/value çiftleri tanımladık.
spring:
  config:
    import:
      - aws-secretsmanager:dev/my-app/database-creds
      - aws-secretsmanager:dev/my-app/oauth-creds
      - optional:aws-secretsmanager:dev/my-app/some-other-creds
Bu çiftlerin herhangi birine @Value ile erişebiliriz. Şöyle yaparız
@SpringBootApplication
public class SpringBootAwsSecretsApplication implements CommandLineRunner {

  @Value("${dbUser}")
  private String dbUser;

  @Value("${dbPassword}")
  private String dbPassword;
  ...
}
Örnek
Şöyle bir secret yaratalım
aws secretsmanager create-secret 
  --name /secret/db-credential 
  --secret-string '{"dbuser": "user1", "dbpassword": "password"}'
Şöyle yaparız. spring/cloud/config/import ile secret ismi belirtiliyor. Secret içindeki dbuser ve dbpassword değişkenleri kullanılıyor. AWS için kullanılan profile ismi "personal"
# actuator configuration
management:
  endpoints:
    web:
      exposure:
        include:
        - env
spring:
  datasource:
    url: jdbc:mysql://localhost:3306/database
    username: ${dbuser}
    password: ${dbpassword}
  jpa:
    hibernate:
      ddl-auto: create

#  AWS configuration
  cloud:
    aws:
      secretsmanager:
        region: eu-central-1
          
      credentials:
        profile:
          name: personal
  config:
    import:
       - aws-secretsmanager:/secret/db-credential
       - optional:aws-secretsmanager:/secrets/optional-secret
Localstack İle Unit Test
Docker compose ile Localstack'i çalıştırmak için şöyle yaparız
version: "3.8"

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4566:4566"            # LocalStack endpoint
    environment:
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEFAULT_REGION=eu-central-1
    volumes:
      - ./localstack-script:/docker-entrypoint-initaws.d
      - /var/run/docker.sock:/var/run/docker.sock
localstack-script/script.sh içinde şöyle yaparız. awslocal localstack tarafından sağlanır ve aws komutu için bir wrapper
awslocal secretsmanager create-secret 
  --name /secret/spring-boot-app 
  --secret-string '{"property1": "property1-value", "property2": "property2-value"}'

awslocal secretsmanager create-secret 
  --name /secret/db-credential 
  --secret-string '{"dbuser": "user1", "dbpassword": "password"}'
test için application.properties şöyledir
spring:
  cloud:
     aws:
      secretsmanager:
        region: eu-central-1
        endpoint: http://localhost:4566
      credentials:
        access-key: none
        secret-key: none
Test içinde şöyle yaparız. Burada @DynamicPropertySource kullanılmıyor
@SpringBootTest
@AutoConfigureMockMvc
class ApplicationIT {

  @Autowired
  MockMvc mockMvc;

  @Container
  private static LocalStackContainer localStackContainer = 
    new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
      .withCopyFileToContainer(MountableFile.forClasspathResource("script.sh"),
                    "/docker-entrypoint-initaws.d/")
      .withServices(LocalStackContainer.Service.SECRETSMANAGER);

  @BeforeAll
  static void beforeAll() throws IOException, InterruptedException {
    System.setProperty("spring.cloud.aws.secretsmanager.endpoint",
      localStackContainer.getEndpointOverride(
        LocalStackContainer.Service.SECRETSMANAGER).toString());

    System.setProperty("spring.cloud.aws.secretsmanager.region", 
      localStackContainer.getRegion());
    System.setProperty("spring.cloud.aws.credentials.access-key", "none");
    System.setProperty("spring.cloud.aws.credentials.secret-key", "none");
  }
  ...
 
}
   







13 Ocak 2021 Çarşamba

SpringCloud AWS SQS

Giriş
Bir örnek burada

Maven
Şu satırı dahil ederiz
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-aws</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-aws-messaging</artifactId>
</dependency>
Gradle
Şu satırı dahil ederiz
implementation 'io.awspring.cloud:spring-cloud-starter-aws-messaging'
SQS Test
Örnek
Localstack yerine şöyle yaparız
version: "3"

services:
  sqs:
    image: roribio16/alpine-sqs
    ports:
      - '9334:9324'
      - '9335:9325'
    volumes:
      - my-datavolume:/sqs-data
volumes:
  my-datavolume:

Örnek
application.yml dosyasında şöyle yaparız
cloud:
  aws:
    region:
      static: us-east-1  # Region where you create the queue
      auto: false
    credentials:
      access-key: #access key
      secret-key: #acess secret key
QueueMessagingTemplate Sınıfı
Şu satırı dahil ederiz
import org.springframework.cloud.aws.messaging.core.QueueMessagingTemplate;
Açıklaması şöyle
With spring-cloud-starter-aws-messaging we get QueueMessagingTemplate. It is an abstraction to easily publish/receive messages to queue. It also provides apis to configure various converters.

constructor - AmazonSQSAsync 
Örnek
Şöyle yaparız. Burada profile dev ise yerelde çalışan SQS'e bağlanıyor
@Configuration
public class SqsConfig {
  private final int BATCH_SIZE = 10;
  private final String queueEndpoint;

  public SqsConfig(@Value("${queue.endpoint}") String queueEndpoint) {
    this.queueEndpoint = queueEndpoint;
  }

  @Bean
  @Primary
  public AmazonSQSAsync amazonSQSAsync(@Value("${spring.profiles.active}")
    String profile) {
    
    if(profile.equals("dev")){
      return AmazonSQSAsyncClientBuilder
        .standard()
        .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
          queueEndpoint,
          Region.AP_SOUTHEAST_1.id()))
        .build();
    }
    else {
      return AmazonSQSAsyncClientBuilder
        .standard()
        .withRegion(Region.AP_SOUTHEAST_1.id())
        .build();
    }
  }

  @Bean
  public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSQSAsync) {
   return new QueueMessagingTemplate(amazonSQSAsync);
  }
}
Örnek
Şöyle yaparız
import org.springframework.cloud.aws.messaging.core.QueueMessagingTemplate;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.sqs.AmazonSQSAsync;
import com.amazonaws.services.sqs.AmazonSQSAsyncClientBuilder;
@Configuration
public class SqsConfig {

  @Value("${cloud.aws.region.static}")
  private String region;   // region name

  @Value("${cloud.aws.credentials.access-key}")
  private String awsAccessKey;   // aws access key.

  @Value("${cloud.aws.credentials.secret-key}")
  private String awsSecretKey; // aws secret key

  @Bean
  public QueueMessagingTemplate queueMessagingTemplate() {
    return new QueueMessagingTemplate(amazonSQSAsync());
  }

  @Bean
  public AmazonSQSAsync amazonSQSAsync() {
    return AmazonSQSAsyncClientBuilder
      .standard()
      .withRegion(region) //region name wich aquired by application.yml file
      .withCredentials(new AWSStaticCredentialsProvider(
       new BasicAWSCredentials(awsAccessKey, awsSecretKey))) // access key and secret key
      .build();
  }
}
convertAndSend metodu
Belirtilen kuyruk ismine Java nesnesini gönderir

Örnek
Şöyle yaparız
Slf4j
@Service
public class QueueServiceImpl implements QueueService {
  private final QueueMessagingTemplate queueMessagingTemplate;
  private final String taskQueue;

  public QueueServiceImpl(QueueMessagingTemplate queueMessagingTemplate,
                         @Value("${queue.task}") String taskQueue) {
    this.queueMessagingTemplate = queueMessagingTemplate;
    this.taskQueue = taskQueue;
  }

  @Override
  public void publishTask(Task task) {

   task.setId(UUID.randomUUID().toString());
   log.info("Publishing task to queue {}", task);

   queueMessagingTemplate.convertAndSend(taskQueue, task);
  }
}
Örnek
Şöyle yaparız. @SqsListener ile sadece String mesaj okuyoruz
import com.aws.sqs.model.Message;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.aws.messaging.core.QueueMessagingTemplate;
import org.springframework.cloud.aws.messaging.listener.SqsMessageDeletionPolicy;
import org.springframework.cloud.aws.messaging.listener.annotation.SqsListener;

@RestController
@RequestMapping(value = "/sqs")
public class SqsController {
  //queue name
  private static final String QUEUE = "my-queue";

  @Autowired
  private QueueMessagingTemplate queueMessagingTemplate;

  @PostMapping(value = "/send")
  @ResponseStatus(code = HttpStatus.CREATED)
  public void sendMessageToSqs(@RequestBody final Message message) {
    queueMessagingTemplate.convertAndSend(QUEUE, message); //send to queue
  }

  // @SqsListener listens to the message from the queue.
  @SqsListener(value = QUEUE, deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
  public void getMessageFromSqs(Message message, @Header("MessageId") String messageId) {
    logger.info("Received message= {} with messageId= {}", message.toString(), messageId);
  }
}
Örnek
Açıklaması şöyle
... we need to let SQS know how it could convert the receiving strings to object. So we add a bean of QueueMessageHandlerFactory.

Şöyle yaparız. @SqsListener Task tipinden Java nesnesi okuyoruz
@Bean
public QueueMessageHandlerFactory queueMessageHandlerFactory(
  ObjectMapper objectMapper, 
  AmazonSQSAsync amazonSQSAsync) {
    
    MappingJackson2MessageConverter messageConverter 
      = new MappingJackson2MessageConverter();
    messageConverter.setObjectMapper(objectMapper);
    messageConverter.setStrictContentTypeMatch(false);

    QueueMessageHandlerFactory factory = new QueueMessageHandlerFactory();
    factory.setAmazonSqs(amazonSQSAsync);

    List<HandlerMethodArgumentResolver> resolvers = List.of(
      new PayloadMethodArgumentResolver(messageConverter,null, false));
   factory.setArgumentResolvers(resolvers);

   return factory;
 }
}

@Service
@Slf4j
public class TaskProcessorImpl implements TaskProcessor {
  @Override
  @SqsListener(value = "dev-task.std",deletionPolicy = SqsMessageDeletionPolicy.ALWAYS)
  @MessageMapping
  public void process(Task task) {
    log.info("Processing task {}", task);
  }
}