Oinone引入搜索引擎(增强模型)

场景描述

在碰到大数据量并且需要全文检索的场景,我们在分布式架构中基本会架设ElasticSearch来作为一个常规解决方案。在oinone体系中增强模型就是应对这类场景,其背后也是整合了ElasticSearch;

使用前你应该

  • 了解ElasticSearch,包括不限于:Index(索引)、分词、Node(节点)、Document(文档)、Shards(分片) & Replicas(副本)。参考官方网站:https://www.elastic.co/cn/
  • 有一个可用的ElasticSearch环境(本地项目能引用到)

前置约束

增强模型增量依赖数据变更实时消息,因此确保项目的event是开启的,mq配置正确。

项目引入搜索步骤

1、boot工程加入相关依赖包

  • boot工程需要指定ES客户端包版本,不指定版本会隐性依赖顶层spring-boot依赖管理指定的低版本
  • boot工程加入pamris-channel的工程依赖
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>8.4.1</version>
</dependency>
<dependency>
    <groupId>jakarta.json</groupId>
    <artifactId>jakarta.json-api</artifactId>
    <version>2.1.1</version>
</dependency>

<dependency>
    <groupId>pro.shushi.pamirs.core</groupId>
    <artifactId>pamirs-sql-record-core</artifactId>
</dependency>
<dependency>
    <groupId>pro.shushi.pamirs.core</groupId>
    <artifactId>pamirs-channel-core</artifactId>
</dependency>

2、api工程加入相关依赖包

在XXX-api中增加入pamirs-channel-api的依赖

<dependency>
    <groupId>pro.shushi.pamirs.core</groupId>
    <artifactId>pamirs-channel-api</artifactId>
</dependency>

3、yml文件配置

在pamirs-demo-boot的application-dev.yml文件中增加配置pamirs.boot.modules增加channel,即在启动模块中增加channel模块。同时注意es的配置,是否跟es的服务一致

pamirs:
  record:
    sql:
      #改成自己本地路径(或服务器路径)
      store: /Users/oinone/record
  boot:
    modules:
      - channel
      ## 确保也安装了sql_record
      - sql_record
  channel:
    packages:
      # 增强模型扫描包配置
      - com.xxx.xxx
  elastic:
    url: 127.0.0.1:9200

4、项目的模块增加模块依赖

XXXModule增加对ChannelModule的依赖

@Module(dependencies = {ChannelModule.MODULE_MODULE})

5、增加增强模型(举例)

package pro.shushi.pamirs.demo.api.enhance;

import pro.shushi.pamirs.channel.enmu.IncrementEnum;
import pro.shushi.pamirs.channel.meta.Enhance;
import pro.shushi.pamirs.channel.meta.EnhanceModel;
import pro.shushi.pamirs.demo.api.model.ShardingModel;
import pro.shushi.pamirs.meta.annotation.Model;
import pro.shushi.pamirs.meta.enmu.ModelTypeEnum;

@Model(displayName = "测试EnhanceModel")
@Model.model(ShardingModelEnhance.MODEL_MODEL)
@Model.Advanced(type = ModelTypeEnum.PROXY, inherited = {EnhanceModel.MODEL_MODEL})
@Enhance(shards = "3", replicas = "1", reAlias = true,increment= IncrementEnum.OPEN)
public class ShardingModelEnhance extends ShardingModel {
    public static final String MODEL_MODEL="demo.ShardingModelEnhance";
}

6、重启系统看效果

1、进入【传输增强模型】应用,访问增强模型列表我们会发现一条记录,并点击【全量同步】初始化ES,并全量dump数据
Oinone引入搜索引擎(增强模型)

2、再次回到Demo应用,进入增强模型页面,可以正常访问并进增删改查操作
Oinone引入搜索引擎(增强模型)

个性化dump逻辑

通常dump逻辑是有个性化需求,那么我们可以重写模型的synchronize方法,函数重写特性在“面向对象-继承与多态”部分中已经有详细介绍。

重写ShardingModelEnhance模型的synchronize方法

重写后,如果针对老数据记录需要把新增的字段都自动填充,可以进入【传输增强模型】应用,访问增强模型列表,找到对应的记录并点击【全量同步】

package pro.shushi.pamirs.demo.api.enhance;

import pro.shushi.pamirs.channel.enmu.IncrementEnum;
import pro.shushi.pamirs.channel.meta.Enhance;
import pro.shushi.pamirs.channel.meta.EnhanceModel;
import pro.shushi.pamirs.demo.api.model.ShardingModel;
import pro.shushi.pamirs.meta.annotation.Field;
import pro.shushi.pamirs.meta.annotation.Function;
import pro.shushi.pamirs.meta.annotation.Model;
import pro.shushi.pamirs.meta.enmu.FunctionTypeEnum;
import pro.shushi.pamirs.meta.enmu.ModelTypeEnum;

import java.util.List;

@Model(displayName = "测试EnhanceModel")
@Model.model(ShardingModelEnhance.MODEL_MODEL)
@Model.Advanced(type = ModelTypeEnum.PROXY, inherited = {EnhanceModel.MODEL_MODEL})
@Enhance(shards = "3", replicas = "1", reAlias = true,increment= IncrementEnum.OPEN)
public class ShardingModelEnhance extends ShardingModel {
    public static final String MODEL_MODEL="demo.ShardingModelEnhance";

    @Field(displayName = "nick")
    private String nick;

    @Function.Advanced(displayName = "同步数据", type = FunctionTypeEnum.UPDATE)
    @Function(summary = "数据同步函数")
    public List<ShardingModelEnhance> synchronize(List<ShardingModelEnhance> data) {
        for(ShardingModelEnhance shardingModelEnhance:data){
            shardingModelEnhance.setNick(shardingModelEnhance.getName());
        }
        return data;
    }
}

给搜索增加个性化逻辑

如果我们需要在查询方法中增加逻辑,在前面的教程中一般是重写queryPage函数,但对于增强模型我们需要重写的是search函数。

个性化search函数

@Function(
        summary = "搜索函数",
        openLevel = {FunctionOpenEnum.LOCAL, FunctionOpenEnum.REMOTE, FunctionOpenEnum.API}
)
@pro.shushi.pamirs.meta.annotation.Function.Advanced(
        type = {FunctionTypeEnum.QUERY},
        category = FunctionCategoryEnum.QUERY_PAGE,
        managed = true
)
public  Pagination<ShardingModelEnhance> search(Pagination<ShardingModelEnhance> page, IWrapper<ShardingModelEnhance> queryWrapper) {
    System.out.println("您的个性化搜索逻辑");
    // return ((IElasticRetrieve) CommonApiFactory.getApi(IElasticRetrieve.class)).search(page, queryWrapper);
    return ((ElasticSearchApi) CommonApiFactory.getApi(ElasticSearchApi.class)).search(page, queryWrapper);
}

个性化search函数示例

    @Override
    @SuppressWarnings({"rawtypes"})
    public <T> Pagination<T> search(Pagination<T> page, IWrapper<T> queryWrapper) {
        String modelModel = queryWrapper.getModel();
        if (null == modelModel || modelModel.isEmpty()) {
            return page;
        }
        ModelConfig modelCfg = PamirsSession.getContext().getModelConfig(modelModel);
        if (null == modelCfg) {
            return page;
        }
        String rsql = queryWrapper.getOriginRsql();
        if (StringUtils.isBlank(rsql)) {
            rsql = "id>0";
        }
        BoolQuery.Builder queryBuilder = ElasticRSQLHelper.parseRSQL(modelCfg, rsql);
        TermQuery isDeletedTerm = QueryBuilders.term()
                .queryName(IS_DELETED)
                .field(IS_DELETED).value(0)
                .build();
        BoolQuery.Builder builder = QueryBuilders.bool().must(new Query(queryBuilder.build()));
        builder.must(new Query(isDeletedTerm));
        String alias = IndexNaming.aliasByModel(modelModel);
        Query query = new Query(builder.build());
        log.info("{}", query);

        List<Order> orders = Optional.ofNullable(page.getSort()).map(Sort::getOrders).orElse(new ArrayList<>());
        int currentPage = Optional.ofNullable(page.getCurrentPage()).orElse(1);
        Long size = Optional.ofNullable(page.getSize()).orElse(10L);
        int pageSize = size.intValue();
        List<SortOptions> sortOptions = new ArrayList<>();
        if (CollectionUtils.isEmpty(orders)) {
            orders.add(new Order(SortDirectionEnum.DESC, ID));
            orders.add(new Order(SortDirectionEnum.DESC, CREATE_DATE));
        }
        for (Order order : orders) {
            sortOptions.add(new SortOptions.Builder()
                    .field(SortOptionsBuilders.field()
                            .field(order.getField())
                            .order(SortDirectionEnum.DESC.equals(order.getDirection()) ? SortOrder.Desc : SortOrder.Asc)
                            .build())
                    .build());
        }

        SearchRequest request = new SearchRequest.Builder()
                .index(alias)
                .from((currentPage - 1) * pageSize)
                .size(pageSize)
                .sort(sortOptions)
                .query(query)
                .highlight(_builder ->
                        _builder.numberOfFragments(4)
                                .fragmentSize(50)
                                .type(HighlighterType.Unified)
                                .fields("name", HighlightField.of(_fieldBuilder -> _fieldBuilder.preTags(ElasticsearchConstant.HIGH_LIGHT_PREFIX).postTags(ElasticsearchConstant.HIGH_LIGHT_POSTFIX)))
                                .fields("documentNo", HighlightField.of(_fieldBuilder -> _fieldBuilder.preTags(ElasticsearchConstant.HIGH_LIGHT_PREFIX).postTags(ElasticsearchConstant.HIGH_LIGHT_POSTFIX)))
                                .fields("keywords", HighlightField.of(_fieldBuilder -> _fieldBuilder.preTags(ElasticsearchConstant.HIGH_LIGHT_PREFIX).postTags(ElasticsearchConstant.HIGH_LIGHT_POSTFIX))))
                .build();

        SearchResponse<HashMap> response = null;
        try {
            log.info("ES搜索请求参数:{}", request.toString());
            response = elasticsearchClient.search(request, HashMap.class);
        } catch (ElasticsearchException e) {
            log.error("索引异常", e);
            PamirsSession.getMessageHub()
                    .msg(Message.init()
                            .setLevel(InformationLevelEnum.WARN)
                            .msg("索引异常"));
            return page;
        } catch (IOException e) {
            log.error("ElasticSearch运行状态异常", e);
            PamirsSession.getMessageHub()
                    .msg(Message.init()
                            .setLevel(InformationLevelEnum.WARN)
                            .msg("ElasticSearch运行状态异常"));
            return page;
        }

        if (null == response || response.timedOut()) {
            return page;
        }

        HitsMetadata<HashMap> hits = response.hits();
        if (null == hits) {
            return page;
        }

        TotalHits totalHits = hits.total();
        long total = Optional.ofNullable(totalHits).map(TotalHits::value).orElse(0L);

        List<HashMap> dataMapList = Optional.of(hits)
                .map(HitsMetadata<HashMap>::hits)
                .map(hitsMap ->{
                    hitsMap.stream().forEach(highlightForEach -> {
                        highlightForEach.highlight().forEach((key, value) -> {
                            if(highlightForEach.source().containsKey(key)){
                                highlightForEach.source().put(key,value.get(0));
                            }
                        });

                    });
                    return hitsMap;
                })
                .map(List::stream)
                .orElse(Stream.empty())
                .map(Hit::source)
                .collect(Collectors.toList());

        List<T> context = persistenceDataConverter.out(modelModel, dataMapList);

        page.setSize(size);
        page.setTotalElements(total);
        page.setContent(context);
        log.info("ES搜索请求参数返回total,{}", total);
        return page;
    }

Oinone社区 作者:望闲原创文章,如若转载,请注明出处:https://doc.oinone.top/backend/7235.html

访问Oinone官网:https://www.oinone.top获取数式Oinone低代码应用平台体验

Like (0)
望闲's avatar望闲数式管理员
Previous 2024年5月14日 pm5:10
Next 2024年5月14日 pm10:15

相关推荐

  • DsHint(指定数据源)和BatchSizeHint(指定批次数量)

    概述和使用场景 DsHintApi ,强制指定数据源, BatchSizeHintApi ,强制指定查询批量数量 API定义 DsHintApi public static DsHintApi model(String model/**模型编码*/) { // 具体实现 } public DsHintApi(Object dsKey/***数据源名称*/) { // 具体实现 } BatchSizeHintApi public static BatchSizeHintApi use(Integer batchSize) { // 具体实现 } 使用示例 1、【注意】代码中使用 try-with-resources语法; 否则可能会出现数据源错乱 2、DsHintApi使用示例包裹在try里面的所有查询都会强制使用指定的数据源 // 使用方式1: try (DsHintApi dsHintApi = DsHintApi.model(PetItem.MODEL_MODEL)) { List<PetItem> items = demoItemDAO.customSqlDemoItem(); PetShopProxy data2 = data.queryById(); data2.fieldQuery(PetShopProxy::getPetTalents); } // 使用方式2: try (DsHintApi dsHintApi = DsHintApi.use("数据源名称")) { List<PetItem> items = demoItemDAO.customSqlDemoItem(); PetShopProxy data2 = data.queryById(); data2.fieldQuery(PetShopProxy::getPetTalents); } 3、BatchSizeHintApi使用示例包裹在try里面的所有查询都会按照指定的batchSize进行查询 // 查询指定每次查询500跳 try (BatchSizeHintApi batchSizeHintApi = BatchSizeHintApi.use(500)) { PetShopProxy data2 = data.queryById(); data2.fieldQuery(PetShopProxy::getPetTalents); } // 查询指定不分页(batchSize=-1)查询。 请注意,你必须在明确不需要分页查询的情况下使用;如果数据量超大不分页可能会卡死。默认不指定分页数的情况下下平台会进行分页查询 try (BatchSizeHintApi batchSizeHintApi = BatchSizeHintApi.use(-1)) { PetShopProxy data2 = data.queryById(); data2.fieldQuery(PetShopProxy::getPetTalents); }

    2024年5月18日
    2.4K00
  • 字段类型之关系描述的特殊场景(常量关联)

    场景概述 【字段类型之关系与引用】一文中已经描述了各种关系字段的常规写法,还有一些特殊场景如:关系映射中存在常量,或者M2M中间表是大于两个字段构成。 场景描述 1、PetTalent模型增加talentType字段2、PetItem与PetTalent的多对多关系增加talentType(达人类型),3、PetItemRelPetTalent中间表维护petItemId、petTalentId以及talentType,PetDogItem和PetCatItem分别重写petTalents字段,关系中增加常量描述。示意图如下: 实际操作步骤 Step1 新增 TalentTypeEnum package pro.shushi.pamirs.demo.api.enumeration; import pro.shushi.pamirs.meta.annotation.Dict; import pro.shushi.pamirs.meta.common.enmu.BaseEnum; @Dict(dictionary = TalentTypeEnum.DICTIONARY,displayName = "达人类型") public class TalentTypeEnum extends BaseEnum<TalentTypeEnum,Integer> { public static final String DICTIONARY ="demo.TalentTypeEnum"; public final static TalentTypeEnum DOG =create("DOG",1,"狗达人","狗达人"); public final static TalentTypeEnum CAT =create("CAT",2,"猫达人","猫达人"); } Step2 PetTalent模型增加talentType字段 package pro.shushi.pamirs.demo.api.model; import pro.shushi.pamirs.demo.api.enumeration.TalentTypeEnum; import pro.shushi.pamirs.meta.annotation.Field; import pro.shushi.pamirs.meta.annotation.Model; @Model.model(PetTalent.MODEL_MODEL) @Model(displayName = "宠物达人",summary="宠物达人",labelFields ={"name"}) public class PetTalent extends AbstractDemoIdModel{ public static final String MODEL_MODEL="demo.PetTalent"; @Field(displayName = "达人") private String name; @Field(displayName = "达人类型") private TalentTypeEnum talentType; } Step3 修改PetItem的petTalents字段,在关系描述中增加talentType(达人类型) @Field.many2many(relationFields = {"petItemId"},referenceFields = {"petTalentId","talentType"},through = PetItemRelPetTalent.MODEL_MODEL ) @Field.Relation(relationFields = {"id"}, referenceFields = {"id","talentType"}) @Field(displayName = "推荐达人",summary = "推荐该商品的达人们") private List<PetTalent> petTalents; Step4 PetDogItem增加petTalents字段,重写父类PetItem的关系描述 talentType配置为常量,填入枚举的值 增加domain描述用户页面选择的时候自动过滤出特定类型的达人,RSQL用枚举的name @Field(displayName = "推荐达人") @Field.many2many( through = "PetItemRelPetTalent", relationFields = {"petItemId"}, referenceFields = {"petTalentId","talentType"} ) @Field.Relation(relationFields = {"id"}, referenceFields = {"id", "#1#"}, domain = " talentType == DOG") private List<PetTalent> petTalents; Step5 PetCatItem增加petTalents字段,重写父类PetItem的关系描述 talentType配置为常量,填入枚举的值 增加domain描述用户页面选择的时候自动过滤出特定类型的达人,RSQL用枚举的name @Field(displayName = "推荐达人") @Field.many2many( through = "PetItemRelPetTalent", relationFields = {"petItemId"}, referenceFields = {"petTalentId","talentType"} ) @Field.Relation(relationFields = {"id"}, referenceFields = {"id", "#2#"}, domain = " talentType == CAT") private List<PetTalent> petTalents; Step6 PetCatItem增加petTalents字段,many2one关系示例 talentType配置为常量,填入枚举的值 增加domain描述用户页面选择的时候自动过滤出特定类型的达人,RSQL用枚举的name @Model.model(PetPet.MODEL_MODEL) @Model(displayName…

    2024年5月25日
    2.5K00
  • Oinone平台可视化调试工具

    为方便开发者定位问题,我们提供了可视化的调试工具。
    该文档将介绍可视化调试工具的基本使用方法。

    2024年4月13日
    1.9K00
  • 读写分离

    总体介绍 Oinone的读写分离方案是基于Sharding-JDBC的整合方案,要先具备一些Sharding-JDBC的知识。 [Sharding-JDBC] 读写分离依赖于主从复制来同步数据,从库复制数据后,才能通过读写分离策略将读请求分发到从库,实现读写操作的分流,请根据业务需求自行实现主从配置。 配置读写策略 配置 top_demo 模块走读写分离的逻辑数据源 pamirsSharding。 配置数据源。 为 pamirsSharding 配置数据源以及 sharding 规则。 指定需要被sharding-jdbc接管的模块 指定top_demo模块给 Sharding-JDBC 接管,接管逻辑数据源名为 pamirsSharding pamirs: framework: data: ds-map: base: base top_demo: pamirsSharding 配置数据源 pamirs: datasource: pamirsMaster: driverClassName: com.mysql.cj.jdbc.Driver type: com.alibaba.druid.pool.DruidDataSource url: jdbc:mysql://127.0.0.1:3306/61_pamirs_mydemo_master?useSSL=false&allowPublicKeyRetrieval=true&useServerPrepStmts=true&cachePrepStmts=true&useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai&autoReconnect=true&allowMultiQueries=true username: root password: ma123456 initialSize: 5 maxActive: 200 minIdle: 5 maxWait: 60000 timeBetweenEvictionRunsMillis: 60000 testWhileIdle: true testOnBorrow: false testOnReturn: false poolPreparedStatements: true asyncInit: true pamirsSlaver: # 从库数据源配置 driverClassName: com.mysql.cj.jdbc.Driver type: com.alibaba.druid.pool.DruidDataSource url: jdbc:mysql://127.0.0.1:3306/61_pamirs_mydemo_slaver?useSSL=false&allowPublicKeyRetrieval=true&useServerPrepStmts=true&cachePrepStmts=true&useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai&autoReconnect=true&allowMultiQueries=true username: root password: ma123456 initialSize: 5 maxActive: 200 minIdle: 5 maxWait: 60000 timeBetweenEvictionRunsMillis: 60000 testWhileIdle: true testOnBorrow: false testOnReturn: false poolPreparedStatements: true asyncInit: true 配置读写数据源及规则 pamirs: sharding: define: data-sources: pamirsSharding: pamirsMaster # 为逻辑数据源pamirsSharding指向主数据源pamirsMaster。 models: "[trigger.PamirsSchedule]": tables: 0..13 rule: pamirsSharding: actual-ds: # 指定逻辑数据源pamirsSharding代理的数据源为pamirsMaster、pamirsSlaver – pamirsMaster – pamirsSlaver # 以下配置跟sharding-jdbc配置一致 replicaQueryRules: – data-sources: pamirsSharding: primaryDataSourceName: pamirsMaster # 写库数据源 replicaDataSourceNames: – pamirsSlaver # 读库数据源 loadBalancerName: round_robin load-balancers: round_robin: type: ROUND_ROBIN # 读写规则

    2025年5月22日
    1.1K00
  • IWrapper、QueryWrapper和LambdaQueryWrapper使用

    条件更新updateByWrapper 通常我们在更新的时候new一个对象出来在去更新,减少更新的字段 Integer update = new DemoUser().updateByWrapper(new DemoUser().setFirstLogin(Boolean.FALSE), Pops.<DemoUser>lambdaUpdate().from(DemoUser.MODEL_MODEL).eq(IdModel::getId, userId) 使用基础模型的updateById方法更新指定字段的方法: new 一下update对象出来,更新这个对象。 WorkflowUserTask userTaskUp = new WorkflowUserTask(); userTaskUp.setId(userTask.getId()); userTaskUp.setNodeContext(json); userTaskUp.updateById(); 条件删除updateByWrapper public List<T> delete(List<T> data) { List<Long> petTypeIdList = new ArrayList(); for(T item:data){ petTypeIdList.add(item.getId()); } Models.data().deleteByWrapper(Pops.<PetType>lambdaQuery().from(PetType.MODEL_MODEL).in(PetType::getId,petTypeIdList)); return data; } 构造条件查询数据 示例1: LambdaQueryWrapper拼接查询条件 private void queryPetShops() { LambdaQueryWrapper<PetShop> query = Pops.<PetShop>lambdaQuery(); query.from(PetShop.MODEL_MODEL); query.setSortable(Boolean.FALSE); query.orderBy(true, true, PetShop::getId); List<PetShop> petShops2 = new PetShop().queryList(query); System.out.printf(petShops2.size() + ""); } 示例2: IWrapper拼接查询条件 private void queryPetShops() { IWrapper<PetShop> wrapper = Pops.<PetShop>lambdaQuery() .from(PetShop.MODEL_MODEL).eq(PetShop::getId,1L); List<PetShop> petShops4 = new PetShop().queryList(wrapper); System.out.printf(petShops4.size() + ""); } 示例3: QueryWrapper拼接查询条件 private void queryPetShops() { //使用Lambda获取字段名,防止后面改字段名漏改 String nameField = LambdaUtil.fetchFieldName(PetTalent::getName); //使用Lambda获取Clumon名,防止后面改字段名漏改 String nameColumn = PStringUtils.fieldName2Column(nameField); QueryWrapper<PetShop> wrapper2 = new QueryWrapper<PetShop>().from(PetShop.MODEL_MODEL) .eq(nameColumn, "test"); List<PetShop> petShops5 = new PetShop().queryList(wrapper2); System.out.printf(petShops5.size() + ""); } IWrapper转为LambdaQueryWrapper @Function.Advanced(type= FunctionTypeEnum.QUERY) @Function.fun(FunctionConstants.queryPage) @Function(openLevel = {FunctionOpenEnum.API}) public Pagination<PetShopProxy> queryPage(Pagination<PetShopProxy> page, IWrapper<PetShopProxy> queryWrapper) { LambdaQueryWrapper<PetShopProxy> wrapper = ((QueryWrapper<PetShopProxy>) queryWrapper).lambda(); // 非存储字段从QueryData中获取 Map<String, Object> queryData = queryWrapper.getQueryData(); if (null != queryData && !queryData.isEmpty()) { String codes = (String) queryData.get("codes"); if (org.apache.commons.lang3.StringUtils.isNotBlank(codes)) { wrapper.in(PetShopProxy::getCode, codes.split(",")); } } return new PetShopProxy().queryPage(page, wrapper); }

    2024年5月25日
    3.3K00

Leave a Reply

Please Login to Comment